Re: Corosync 2.3 dies randomly

[Angus Salkeld <asalkeld@xxxxxxxxxx>]

On 08/05/13 11:29 -0700, Steven Dake wrote:
> On 05/08/2013 11:23 AM, Robert Parsons wrote:
> > Yes, it's x86_64. The nodes are actually Xen VMs. I am using the 
> > latest libqb from github.
> Copying list so the community can assist.
>
> Angus, do you have any thoughts on why this problem exists or how to 
> assist Robert in further debugging this libqb crash?

What version of libqb are you using?

Can you try build libqb from source too?
https://github.com/asalkeld/libqb

-Angus

> Regards
> -steve
>
> > > I assume your running on x86_64 and not some other architecture. 
> > > Could you verify?
> > >
> > > Typically a bus error occurs because the mmap() operations in 
> > > libqb are failing in some way on x86_64 architectures.  In the 
> > > past, this is because the tmpfs that stores the memory maps is 
> > > not large enough.  Are you using latest master of libqb?
> > >
> > > Regards
> > > -steve
> > >
> > > > strace output
> > > >
> > > > sendmsg(11, {msg_name(16)={sa_family=AF_INET, 
> > > > sin_port=htons(9000), sin_addr=inet_addr("226.94.1.3")}, msg_iov(1)=[{"\376\376\0\0\2\0\"\377w\4\1\n\2w\4\1\n\2\0\n\1\4w\0\0\0\0\0\0\0\0\0"..., 
> > > > 87}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 87
> > > > sendmsg(10, {msg_name(0)=NULL, msg_iov(1)=[{"\376\376\0\0\2\0\"\377w\4\1\n\2w\4\1\n\2\0\n\1\4w\0\0\0\0\0\0\0\0\0"..., 
> > > > 87}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 87
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 710432961}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 710674580}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 711018759}) = 0
> > > > epoll_wait(4, {{EPOLLIN, {u32=2, u64=5813885752995479554}}}, 12, 0) = 1
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 711595469}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 711888254}) = 0
> > > > epoll_wait(4, {{EPOLLIN, {u32=2, u64=5813885752995479554}}}, 
> > > > 12, 257) = 1
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 712421751}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 712715186}) = 0
> > > > recvmsg(9, {msg_name(0)={sa_family=0x7c50 /* AF_??? */, 
> > > > sa_data="\234\217A\177\0\0\20s\234\217A\177\0\0"}, msg_iov(1)=[{"\376\376\0\0\2\0\"\377w\4\1\n\2w\4\1\n\2\0\n\1\4w\0\0\0\0\0\0\0\0\0"..., 
> > > > 10000}], msg_controllen=0, msg_flags=0}, 
> > > > MSG_DONTWAIT|MSG_NOSIGNAL) = 87
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 713490334}) = 0
> > > > epoll_wait(4, {}, 12, 0)                = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 714051544}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 714348259}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 714558026}) = 0
> > > > epoll_wait(4, {}, 12, 0)                = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 715203138}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 715411760}) = 0
> > > > epoll_wait(4, {{EPOLLIN, {u32=1, u64=7284734596711710721}}}, 
> > > > 12, 254) = 1
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 930356426}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 930649677}) = 0
> > > > recvmsg(8, {msg_name(16)={sa_family=AF_INET, 
> > > > sin_port=htons(8999), sin_addr=inet_addr("10.1.4.124")}, msg_iov(1)=[{"\376\376\0\0\5\0\"\377|\4\1\nw\4\1\n\2\0\n\1\4w\0\0\0\0\0\0\0\0\0\0"..., 
> > > > 10000}], msg_controllen=0, msg_flags=0}, 
> > > > MSG_DONTWAIT|MSG_NOSIGNAL) = 42
> > > > sendmsg(11, {msg_name(16)={sa_family=AF_INET, 
> > > > sin_port=htons(9000), sin_addr=inet_addr("10.1.4.121")}, msg_iov(1)=[{"\376\376\0\0\0\0\"\377w\4\1\n\256\0\0\0\10=\0\0\256\0\0\0\0\0\0\0w\4\1\n"..., 
> > > > 74}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 74
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 931677894}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 931933853}) = 0
> > > > epoll_wait(4, {}, 12, 0)                = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 932546611}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 932960908}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 933318531}) = 0
> > > > epoll_wait(4, {}, 12, 0)                = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 933818440}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 934122715}) = 0
> > > > epoll_wait(4, {{EPOLLIN, {u32=1, u64=7284734596711710721}}}, 
> > > > 12, 35) = 1
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 939059907}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 939326101}) = 0
> > > > recvmsg(8, {msg_name(16)={sa_family=AF_INET, 
> > > > sin_port=htons(8999), sin_addr=inet_addr("10.1.4.124")}, msg_iov(1)=[{"\376\376\0\0\1\2\"\377|\4\1\n\2|\4\1\n\2\0\n\1\4|\0\0\0\0\0\0\0\0\0"..., 
> > > > 10000}], msg_controllen=0, msg_flags=0}, 
> > > > MSG_DONTWAIT|MSG_NOSIGNAL) = 1472
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 939958334}) = 0
> > > > epoll_wait(4, {{EPOLLIN, {u32=1, u64=7284734596711710721}}, 
> > > > {EPOLLIN, {u32=3, u64=8736411904714473475}}}, 12, 0) = 2
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 940609120}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 940881911}) = 0
> > > > recvmsg(8, {msg_name(16)={sa_family=AF_INET, 
> > > > sin_port=htons(8999), sin_addr=inet_addr("10.1.4.124")}, msg_iov(1)=[{"\376\376\0\0\1\2\"\377|\4\1\n\2|\4\1\n\2\0\n\1\4|\0\0\0\0\0\0\0\0\0"..., 
> > > > 10000}], msg_controllen=0, msg_flags=0}, 
> > > > MSG_DONTWAIT|MSG_NOSIGNAL) = 617
> > > > sendto(23, "\0", 1, MSG_NOSIGNAL, NULL, 0) = 1
> > > > recvmsg(12, {msg_name(16)={sa_family=AF_INET, 
> > > > sin_port=htons(8999), sin_addr=inet_addr("10.1.4.133")}, msg_iov(1)=[{"\376\376\0\0\0\0\"\377\205\4\1\n\262\0\0\0\20=\0\0\262\0\0\0\0\0\0\0w\4\1\n"..., 
> > > > 10000}], msg_controllen=0, msg_flags=0}, 
> > > > MSG_DONTWAIT|MSG_NOSIGNAL) = 74
> > > > poll([{fd=8, events=POLLIN}], 1, 0)     = 1 ([{fd=8, revents=POLLIN}])
> > > > recvmsg(8, {msg_name(16)={sa_family=AF_INET, 
> > > > sin_port=htons(8999), sin_addr=inet_addr("10.1.4.133")}, msg_iov(1)=[{"\376\376\0\0\5\0\"\377\205\4\1\nw\4\1\n\2\0\n\1\4w\0\0\0\0\0\0\0\0\0\0"..., 
> > > > 10000}], msg_controllen=0, msg_flags=0}, 
> > > > MSG_DONTWAIT|MSG_NOSIGNAL) = 42
> > > > sendmsg(11, {msg_name(16)={sa_family=AF_INET, 
> > > > sin_port=htons(9000), sin_addr=inet_addr("10.1.4.121")}, msg_iov(1)=[{"\376\376\0\0\0\0\"\377w\4\1\n\256\0\0\0\10=\0\0\256\0\0\0\0\0\0\0w\4\1\n"..., 
> > > > 74}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 74
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 943922220}) = 0
> > > > poll([{fd=8, events=POLLIN}], 1, 0)     = 1 ([{fd=8, revents=POLLIN}])
> > > > recvmsg(8, {msg_name(16)={sa_family=AF_INET, 
> > > > sin_port=htons(8999), sin_addr=inet_addr("10.1.4.133")}, msg_iov(1)=[{"\376\376\0\0\1\2\"\377\205\4\1\n\2\205\4\1\n\2\0\n\1\4\205\0\0\0\0\0\0\0\0\0"..., 
> > > > 10000}], msg_controllen=0, msg_flags=0}, 
> > > > MSG_DONTWAIT|MSG_NOSIGNAL) = 1472
> > > > poll([{fd=8, events=POLLIN}], 1, 0)     = 1 ([{fd=8, revents=POLLIN}])
> > > > recvmsg(8, {msg_name(16)={sa_family=AF_INET, 
> > > > sin_port=htons(8999), sin_addr=inet_addr("10.1.4.133")}, msg_iov(1)=[{"\376\376\0\0\1\2\"\377\205\4\1\n\2\205\4\1\n\2\0\n\1\4\205\0\0\0\0\0\0\0\0\0"..., 
> > > > 10000}], msg_controllen=0, msg_flags=0}, 
> > > > MSG_DONTWAIT|MSG_NOSIGNAL) = 584
> > > > sendto(23, "\0", 1, MSG_NOSIGNAL, NULL, 0) = 1
> > > > poll([{fd=8, events=POLLIN}], 1, 0)     = 0 (Timeout)
> > > > poll([{fd=9, events=POLLIN}], 1, 0)     = 0 (Timeout)
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 946686072}) = 0
> > > > sendmsg(11, {msg_name(16)={sa_family=AF_INET, 
> > > > sin_port=htons(9000), sin_addr=inet_addr("10.1.4.121")}, msg_iov(1)=[{"\376\376\0\0\0\0\"\377w\4\1\n\262\0\0\0\21=\0\0\262\0\0\0\0\0\0\0w\4\1\n"..., 
> > > > 74}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 74
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 952769552}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 952985732}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 953217383}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 953489345}) = 0
> > > > epoll_wait(4, {}, 12, 0)                = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 954019395}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 954314404}) = 0
> > > > epoll_wait(4, {{EPOLLIN, {u32=3, u64=8736411904714473475}}}, 
> > > > 12, 586) = 1
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 954907616}) = 0
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 955138096}) = 0
> > > > recvmsg(12, {msg_name(16)={sa_family=AF_INET, 
> > > > sin_port=htons(8999), sin_addr=inet_addr("10.1.4.133")}, msg_iov(1)=[{"\376\376\0\0\0\0\"\377\205\4\1\n\262\0\0\0\31=\0\0\262\0\0\0\0\0\0\0w\4\1\n"..., 
> > > > 10000}], msg_controllen=0, msg_flags=0}, 
> > > > MSG_DONTWAIT|MSG_NOSIGNAL) = 74
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 955849968}) = 0
> > > > poll([{fd=8, events=POLLIN}], 1, 0)     = 0 (Timeout)
> > > > poll([{fd=9, events=POLLIN}], 1, 0)     = 0 (Timeout)
> > > > clock_gettime(CLOCK_MONOTONIC, {57355, 956746383}) = 0
> > > > --- SIGBUS (Bus error) @ 0 (0) ---
> > > > Process 30095 detached
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > corosync-blackbox output
> > > >
> > > >
> > > > root@krusty:~# corosync-blackbox
> > > > Failed to initialize the cmap API. Error CS_ERR_LIBRARY
> > > > Failed to initialize the cmap API. Error CS_ERR_LIBRARY
> > > > Dumping the contents of /var/lib/corosync/fdata
> > > > [debug] shm size:8388608; real_size:8388608; rb->word_size:2097152
> > > > [debug] read total of: 8388620
> > > > Ringbuffer:
> > > > ->NORMAL
> > > > ->write_pt [673]
> > > > ->read_pt [0]
> > > > ->size [2097152 words]
> > > > =>free [8385912 bytes]
> > > > =>used [2692 bytes]
> > > > debug   May 07 15:26:52 handle_new_connection(476):2147483648: 
> > > > IPC credentials authenticated (13438-14116-18)
> > > > debug   May 07 15:26:52 qb_ipcs_shm_connect(294):9: connecting 
> > > > to client [14116]
> > > > debug   May 07 15:26:52 qb_rb_open_2(226):9: shm size:zd; 
> > > > real_size:zd; rb->word_size:1048576
> > > > debug   May 07 15:26:52 qb_rb_open_2(226):9: shm size:zd; 
> > > > real_size:zd; rb->word_size:1048576
> > > > debug   May 07 15:26:52 qb_rb_open_2(226):9: shm size:zd; 
> > > > real_size:zd; rb->word_size:1048576
> > > > debug   May 07 15:26:52 cs_ipcs_connection_created(269):8: 
> > > > connection created
> > > > debug   May 07 15:26:52 cmap_lib_init_fn(306):9: lib_init_fn: 
> > > > conn=0x7feb08b0ea00
> > > > debug   May 07 15:26:52 
> > > > qb_ipcs_dispatch_connection_request(723):9: HUP conn 
> > > > (13438-14116-18)
> > > > debug   May 07 15:26:52 qb_ipcs_disconnect(565):9: 
> > > > qb_ipcs_disconnect(13438-14116-18) state:2
> > > > debug   May 07 15:26:52 _del(117):9: epoll_ctl(del): Bad file 
> > > > descriptor (9)
> > > > debug   May 07 15:26:52 cs_ipcs_connection_closed(414):8: 
> > > > cs_ipcs_connection_closed()
> > > > debug   May 07 15:26:52 cmap_lib_exit_fn(325):9: exit_fn for 
> > > > conn=0x7feb08b0ea00
> > > > debug   May 07 15:26:52 cs_ipcs_connection_destroyed(387):8: 
> > > > cs_ipcs_connection_destroyed()
> > > > trace   May 07 15:26:52 qb_rb_close(279):9: ENTERING qb_rb_close()
> > > > trace   May 07 15:26:52 my_posix_sem_destroy(91):9: ENTERING 
> > > > my_posix_sem_destroy()
> > > > debug   May 07 15:26:52 qb_rb_close(290):9: Free'ing 
> > > > ringbuffer: /dev/shm/qb-cmap-response-13438-14116-18-header
> > > > trace   May 07 15:26:52 qb_rb_close(279):9: ENTERING qb_rb_close()
> > > > trace   May 07 15:26:52 my_posix_sem_destroy(91):9: ENTERING 
> > > > my_posix_sem_destroy()
> > > > debug   May 07 15:26:52 qb_rb_close(290):9: Free'ing 
> > > > ringbuffer: /dev/shm/qb-cmap-event-13438-14116-18-header
> > > > trace   May 07 15:26:52 qb_rb_close(279):9: ENTERING qb_rb_close()
> > > > trace   May 07 15:26:52 my_posix_sem_destroy(91):9: ENTERING 
> > > > my_posix_sem_destroy()
> > > > debug   May 07 15:26:52 qb_rb_close(290):9: Free'ing 
> > > > ringbuffer: /dev/shm/qb-cmap-request-13438-14116-18-header
> > > > debug   May 07 15:26:52 handle_new_connection(476):2147483648: 
> > > > IPC credentials authenticated (13438-14118-18)
> > > > debug   May 07 15:26:52 qb_ipcs_shm_connect(294):9: connecting 
> > > > to client [14118]
> > > > debug   May 07 15:26:52 qb_rb_open_2(226):9: shm size:zd; 
> > > > real_size:zd; rb->word_size:1048576
> > > > debug   May 07 15:26:52 qb_rb_open_2(226):9: shm size:zd; 
> > > > real_size:zd; rb->word_size:1048576
> > > > debug   May 07 15:26:52 qb_rb_open_2(226):9: shm size:zd; 
> > > > real_size:zd; rb->word_size:1048576
> > > > debug   May 07 15:26:52 cs_ipcs_connection_created(269):8: 
> > > > connection created
> > > > debug   May 07 15:26:52 cmap_lib_init_fn(306):9: lib_init_fn: 
> > > > conn=0x7feb08b0ea00
> > > > ERROR: qb_rb_chunk_read failed: Connection timed out
> > > > [trace] ENTERING qb_rb_close()
> > > > [debug] Free'ing ringbuffer: /dev/shm/qb-create_from_file-header
> > > >
> > > >
> > > >
> > > >
> > > > core dump
> > > >
> > > > Reading symbols from /usr/sbin/corosync...done.
> > > > [New LWP 30095]
> > > > [New LWP 30096]
> > > >
> > > > warning: Can't read pathname for load map: Input/output error.
> > > > [Thread debugging using libthread_db enabled]
> > > > Using host libthread_db library 
> > > > "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > > > Core was generated by `corosync'.
> > > > Program terminated with signal 7, Bus error.
> > > > #0  0x00007f418e4ea823 in qb_rb_chunk_alloc (rb=0x7f418f9c0d20, 
> > > > len=<optimized out>) at ringbuffer.c:456
> > > > 456             rb->shared_data[write_pt] = 0;
> > > >
> > > > (gdb) thread apply all bt
> > > >
> > > > Thread 2 (Thread 0x7f418bf6f700 (LWP 30096)):
> > > > #0  0x00007f418e2cdfd0 in sem_wait () from 
> > > > /lib/x86_64-linux-gnu/libpthread.so.0
> > > > #1  0x00007f418e4f45f3 in qb_logt_worker_thread 
> > > > (data=<optimized out>) at log_thread.c:71
> > > > #2  0x00007f418e2c7e9a in start_thread () from 
> > > > /lib/x86_64-linux-gnu/libpthread.so.0
> > > > #3  0x00007f418dff3cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
> > > > #4  0x0000000000000000 in ?? ()
> > > >
> > > > Thread 1 (Thread 0x7f418ed96700 (LWP 30095)):
> > > > #0  0x00007f418e4ea823 in qb_rb_chunk_alloc (rb=0x7f418f9c0d20, 
> > > > len=<optimized out>) at ringbuffer.c:456
> > > > #1  0x00007f418e4f4bb9 in _blackbox_vlogger (target=<optimized 
> > > > out>, cs=0x7f418fa3e4d8, timestamp=1368027286,
> > > >    ap=0x7fff3408ebf8) at log_blackbox.c:74
> > > > #2  0x00007f418e4f2dea in qb_log_real_va_ (cs=0x7f418fa3e4d8, 
> > > > ap=0x7fff3408ee88) at log.c:184
> > > > #3  0x00007f418edbdf00 in _logsys_log_printf (level=<optimized 
> > > > out>, subsys=<optimized out>,
> > > >    function_name=<optimized out>, file_name=<optimized out>, 
> > > > file_line=<optimized out>, format=<optimized out>)
> > > >    at main.c:898
> > > > #4  0x00007f418e95f7bd in messages_free (token_aru=<optimized 
> > > > out>, instance=0x7f418b732010) at totemsrp.c:2462
> > > > #5  message_handler_orf_token (instance=0x7f418b732010, 
> > > > msg=<optimized out>, endian_conversion_needed=<optimized out>,
> > > >    msg_len=<optimized out>) at totemsrp.c:3611
> > > > #6  0x00007f418e9617f7 in message_handler_orf_token 
> > > > (instance=<optimized out>, msg=<optimized out>,
> > > >    msg_len=<optimized out>, 
> > > > endian_conversion_needed=<optimized out>) at totemsrp.c:3532
> > > > #7  0x00007f418e95bcc1 in rrp_deliver_fn 
> > > > (context=0x7f418f9ce850, msg=0x7f418fa0f868, msg_len=70) at 
> > > > totemrrp.c:1783
> > > > #8  0x00007f418e956ad2 in net_deliver_fn (fd=<optimized out>, 
> > > > revents=<optimized out>, data=0x7f418fa0f800)
> > > >    at totemudp.c:521
> > > > #9  0x00007f418e4ec05f in _poll_dispatch_and_take_back_ 
> > > > (item=<optimized out>, p=<optimized out>) at loop_poll.c:98
> > > > #10 0x00007f418e4ebbd7 in qb_loop_run_level 
> > > > (level=0x7f418f9c7268) at loop.c:43
> > > > #11 qb_loop_run (lp=<optimized out>) at loop.c:204
> > > > #12 0x00007f418edacbe5 in main (argc=<optimized out>, 
> > > > argv=<optimized out>, envp=<optimized out>) at main.c:1309
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > syslog
> > > >
> > > >
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo 
> > > > message from cluster node 167838851
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo 
> > > > message[167838851]: votes: 1, expected: 17 flags: 1
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] flags: 
> > > > quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No 
> > > > QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] 
> > > > total_votes=8, expected_votes=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838839 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838841 state=2, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838842 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838844 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838846 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838849 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838850 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838851 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838853 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] quorum lost, 
> > > > blocking activity
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo 
> > > > message from cluster node 167838851
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo 
> > > > message[0]: votes: 0, expected: 0 flags: 0
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo 
> > > > message from cluster node 167838853
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo 
> > > > message[167838853]: votes: 1, expected: 17 flags: 1
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] flags: 
> > > > quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No 
> > > > QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] 
> > > > total_votes=8, expected_votes=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838839 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838841 state=2, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838842 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838844 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838846 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838849 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838850 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838851 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838853 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo 
> > > > message from cluster node 167838853
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo 
> > > > message[0]: votes: 0, expected: 0 flags: 0
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo 
> > > > message from cluster node 167838839
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo 
> > > > message[167838839]: votes: 1, expected: 17 flags: 1
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] flags: 
> > > > quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No 
> > > > QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] 
> > > > total_votes=8, expected_votes=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838839 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838841 state=2, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838842 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838844 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838846 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838849 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838850 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838851 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838853 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo 
> > > > message from cluster node 167838839
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo 
> > > > message[0]: votes: 0, expected: 0 flags: 0
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo 
> > > > message from cluster node 167838842
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo 
> > > > message[167838842]: votes: 1, expected: 17 flags: 1
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] flags: 
> > > > quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No 
> > > > QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] 
> > > > total_votes=8, expected_votes=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838839 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838841 state=2, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838842 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838844 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838846 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838849 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838850 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838851 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838853 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo 
> > > > message from cluster node 167838842
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo 
> > > > message[0]: votes: 0, expected: 0 flags: 0
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo 
> > > > message from cluster node 167838844
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo 
> > > > message[167838844]: votes: 1, expected: 17 flags: 1
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] flags: 
> > > > quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No 
> > > > QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] 
> > > > total_votes=8, expected_votes=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838839 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838841 state=2, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838842 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838844 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838846 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838849 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838850 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838851 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838853 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo 
> > > > message from cluster node 167838844
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo 
> > > > message[0]: votes: 0, expected: 0 flags: 0
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo 
> > > > message from cluster node 167838846
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo 
> > > > message[167838846]: votes: 1, expected: 17 flags: 1
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] flags: 
> > > > quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No 
> > > > QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] 
> > > > total_votes=8, expected_votes=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838839 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838841 state=2, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838842 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838844 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838846 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 
> > > > 167838849 state=1, votes=1, expected=17
> > > > May  8 10:26:23 krusty rsyslogd-2177: imuxsock begins to drop 
> > > > messages from pid 30095 due to rate-limiting
> > > > May  8 10:26:23 krusty crmd[30689]:   notice: 
> > > > pcmk_quorum_notification: Membership 47416: quorum lost (8)
> > > > May  8 10:26:24 krusty crmd[30689]:   notice: 
> > > > pcmk_quorum_notification: Membership 47420: quorum acquired (9)
> > > > May  8 10:26:25 krusty crmd[30689]:   notice: 
> > > > corosync_node_name: Unable to get node name for nodeid 
> > > > 167838841
> > > > May  8 10:27:37 krusty stonith-ng[30687]:   notice: 
> > > > unpack_config: On loss of CCM Quorum: Ignore
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=quimby - 
> > > > this is rarely intended
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=todd - 
> > > > this is rarely intended
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=rod - 
> > > > this is rarely intended
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=maude - 
> > > > this is rarely intended
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=krusty - 
> > > > this is rarely intended
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=itchy - 
> > > > this is rarely intended
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with 
> > > > uname=sideshowbob - this is rarely intended
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=willie - 
> > > > this is rarely intended
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=ned - 
> > > > this is rarely intended
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=moe - 
> > > > this is rarely intended
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=nelson - 
> > > > this is rarely intended
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=scratchy 
> > > > - this is rarely intended
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=fattony 
> > > > - this is rarely intended
> > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=barney - 
> > > > this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:   notice: 
> > > > unpack_config: On loss of CCM Quorum: Ignore
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=quimby - 
> > > > this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=todd - 
> > > > this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=rod - 
> > > > this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=maude - 
> > > > this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=krusty - 
> > > > this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=itchy - 
> > > > this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with 
> > > > uname=sideshowbob - this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=willie - 
> > > > this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=ned - 
> > > > this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=moe - 
> > > > this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=nelson - 
> > > > this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=scratchy 
> > > > - this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=fattony 
> > > > - this is rarely intended
> > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=barney - 
> > > > this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:   notice: 
> > > > unpack_config: On loss of CCM Quorum: Ignore
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=quimby - 
> > > > this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=todd - 
> > > > this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=rod - 
> > > > this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=maude - 
> > > > this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=krusty - 
> > > > this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=itchy - 
> > > > this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with 
> > > > uname=sideshowbob - this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=willie - 
> > > > this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=ned - 
> > > > this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=moe - 
> > > > this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=nelson - 
> > > > this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=scratchy 
> > > > - this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=fattony 
> > > > - this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > create_node: Detected multiple node entries with uname=barney - 
> > > > this is rarely intended
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > unpack_rsc_location: No resource 
> > > > (con=cli-standby-res_IPaddr2_NXTProduction, 
> > > > rsc=res_IPaddr2_NXTProduction)
> > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: 
> > > > unpack_rsc_location: No resource 
> > > > (con=cli-prefer-res_IPaddr2_NXTProduction, 
> > > > rsc=res_IPaddr2_NXTProduction)
> > > > May  8 10:34:46 krusty pacemakerd[30683]:    error: 
> > > > cfg_connection_destroy: Connection destroyed
> > > > May  8 10:34:46 krusty pacemakerd[30683]:   notice: 
> > > > pcmk_shutdown_worker: Shuting down Pacemaker
> > > > May  8 10:34:46 krusty pacemakerd[30683]:   notice: stop_child: 
> > > > Stopping crmd: Sent -15 to process 30689
> > > > May  8 10:34:46 krusty crmd[30689]:   notice: crm_shutdown: 
> > > > Requesting shutdown, upper limit is 1200000ms
> > > > May  8 10:34:46 krusty pacemakerd[30683]:    error: 
> > > > cpg_connection_destroy: Connection destroyed
> > > > May  8 10:34:46 krusty stonith-ng[30687]:    error: 
> > > > pcmk_cpg_dispatch: Connection to the CPG API failed: 2
> > > > May  8 10:34:46 krusty stonith-ng[30687]:    error: 
> > > > stonith_peer_ais_destroy: AIS connection terminated
> > > > May  8 10:34:46 krusty crmd[30689]:    error: crm_ipc_read: 
> > > > Connection to stonith-ng failed
> > > > May  8 10:34:46 krusty crmd[30689]:    error: 
> > > > mainloop_gio_callback: Connection to stonith-ng[0x1cce4e0] 
> > > > closed (I/O condition=17)
> > > > May  8 10:34:46 krusty crmd[30689]:     crit: 
> > > > tengine_stonith_connection_destroy: Fencing daemon connection 
> > > > failed
> > > > May  8 10:34:46 krusty attrd[30688]:    error: 
> > > > pcmk_cpg_dispatch: Connection to the CPG API failed: 2
> > > > May  8 10:34:46 krusty attrd[30688]:     crit: 
> > > > attrd_ais_destroy: Lost connection to Corosync service!
> > > > May  8 10:34:46 krusty attrd[30688]:   notice: main: Exiting...
> > > > May  8 10:34:46 krusty attrd[30688]:   notice: main: 
> > > > Disconnecting client 0x22e9eb0, pid=30689...
> > > > May  8 10:34:46 krusty attrd[30688]:    error: 
> > > > attrd_cib_connection_destroy: Connection to the CIB 
> > > > terminated...
> > > > May  8 10:34:46 krusty cib[30686]:    error: pcmk_cpg_dispatch: 
> > > > Connection to the CPG API failed: 2
> > > > May  8 10:34:46 krusty cib[30686]:    error: cib_ais_destroy: 
> > > > Corosync connection lost!  Exiting.
> > > > May  8 10:34:47 krusty crmd[30689]:    error: 
> > > > te_connect_stonith: Sign-in failed: triggered a retry
> > > > May  8 10:34:47 krusty crmd[30689]:    error: crm_ipc_read: 
> > > > Connection to cib_shm failed
> > > > May  8 10:34:47 krusty crmd[30689]:    error: 
> > > > mainloop_gio_callback: Connection to cib_shm[0x1baf020] closed 
> > > > (I/O condition=17)
> > > > May  8 10:34:47 krusty crmd[30689]:    error: 
> > > > crmd_cib_connection_destroy: Connection to the CIB 
> > > > terminated...
> > > > May  8 10:34:47 krusty crmd[30689]:    error: 
> > > > crmd_quorum_destroy: connection terminated
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > discuss mailing list
> > > > discuss@xxxxxxxxxxxx
> > > > http://lists.corosync.org/mailman/listinfo/discuss
_______________________________________________
discuss mailing list
discuss@xxxxxxxxxxxx
http://lists.corosync.org/mailman/listinfo/discuss