Re: Corosync 2.3 dies randomly

[Angus Salkeld <asalkeld@xxxxxxxxxx>]

On 08/05/13 20:10 -0500, Rob Parsons wrote:
> I'm using libqb pulled from github on May 4th.

One quick thing to check is the location of your shared memory
I use travis ci for libqb and travis uses ubuntu vm's and I
know I had to do a workaround for the shared memory location
being moved from /dev/shm to /run/shm.

See: https://github.com/asalkeld/libqb/blob/master/.travis.yml

I'd suggest have a look at the output of:
mount | grep shm
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)

df -h | grep shm
tmpfs                    3.9G  2.9M  3.9G   1% /dev/shm

and see if you need to run that workaround. (libqb tries /dev/shm
first).

-Angus

> On Wed, May 8, 2013 at 6:45 PM, Angus Salkeld <asalkeld@xxxxxxxxxx> wrote:
>
> > On 08/05/13 11:29 -0700, Steven Dake wrote:
> >
> > > On 05/08/2013 11:23 AM, Robert Parsons wrote:
> > >
> > > > Yes, it's x86_64. The nodes are actually Xen VMs. I am using the latest
> > > > libqb from github.
> > > >
> > > >  Copying list so the community can assist.
> > >
> > > Angus, do you have any thoughts on why this problem exists or how to
> > > assist Robert in further debugging this libqb crash?
> >
> > What version of libqb are you using?
> >
> > Can you try build libqb from source too?
> > https://github.com/asalkeld/**libqb <https://github.com/asalkeld/libqb>
> >
> > -Angus
> >
> >
> >
> > > Regards
> > > -steve
> > >
> > >
> > > > > I assume your running on x86_64 and not some other architecture. Could
> > > > > you verify?
> > > > >
> > > > > Typically a bus error occurs because the mmap() operations in libqb are
> > > > > failing in some way on x86_64 architectures.  In the past, this is because
> > > > > the tmpfs that stores the memory maps is not large enough.  Are you using
> > > > > latest master of libqb?
> > > > >
> > > > > Regards
> > > > > -steve
> > > > >
> > > > >
> > > > > > strace output
> > > > > >
> > > > > > sendmsg(11, {msg_name(16)={sa_family=AF_**INET, sin_port=htons(9000),
> > > > > > sin_addr=inet_addr("226.94.1.**3")}, msg_iov(1)=[{"\376\376\0\0\2\**
> > > > > > 0\"\377w\4\1\n\2w\4\1\n\2\0\n\**1\4w\0\0\0\0\0\0\0\0\0"..., 87}],
> > > > > > msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 87
> > > > > > sendmsg(10, {msg_name(0)=NULL, msg_iov(1)=[{"\376\376\0\0\2\**
> > > > > > 0\"\377w\4\1\n\2w\4\1\n\2\0\n\**1\4w\0\0\0\0\0\0\0\0\0"..., 87}],
> > > > > > msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 87
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 710432961}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 710674580}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 711018759}) = 0
> > > > > > epoll_wait(4, {{EPOLLIN, {u32=2, u64=5813885752995479554}}}, 12, 0) = 1
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 711595469}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 711888254}) = 0
> > > > > > epoll_wait(4, {{EPOLLIN, {u32=2, u64=5813885752995479554}}}, 12, 257)
> > > > > > = 1
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 712421751}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 712715186}) = 0
> > > > > > recvmsg(9, {msg_name(0)={sa_family=0x7c50 /* AF_??? */,
> > > > > > sa_data="\234\217A\177\0\0\**20s\234\217A\177\0\0"},
> > > > > > msg_iov(1)=[{"\376\376\0\0\2\**0\"\377w\4\1\n\2w\4\1\n\2\0\n\**1\4w\0\0\0\0\0\0\0\0\0"...,
> > > > > > 10000}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 87
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 713490334}) = 0
> > > > > > epoll_wait(4, {}, 12, 0)                = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 714051544}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 714348259}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 714558026}) = 0
> > > > > > epoll_wait(4, {}, 12, 0)                = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 715203138}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 715411760}) = 0
> > > > > > epoll_wait(4, {{EPOLLIN, {u32=1, u64=7284734596711710721}}}, 12, 254)
> > > > > > = 1
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 930356426}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 930649677}) = 0
> > > > > > recvmsg(8, {msg_name(16)={sa_family=AF_**INET, sin_port=htons(8999),
> > > > > > sin_addr=inet_addr("10.1.4.**124")}, msg_iov(1)=[{"\376\376\0\0\5\**
> > > > > > 0\"\377|\4\1\nw\4\1\n\2\0\n\1\**4w\0\0\0\0\0\0\0\0\0\0"..., 10000}],
> > > > > > msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 42
> > > > > > sendmsg(11, {msg_name(16)={sa_family=AF_**INET, sin_port=htons(9000),
> > > > > > sin_addr=inet_addr("10.1.4.**121")}, msg_iov(1)=[{"\376\376\0\0\0\**
> > > > > > 0\"\377w\4\1\n\256\0\0\0\10=\**0\0\256\0\0\0\0\0\0\0w\4\1\n".**..,
> > > > > > 74}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 74
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 931677894}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 931933853}) = 0
> > > > > > epoll_wait(4, {}, 12, 0)                = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 932546611}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 932960908}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 933318531}) = 0
> > > > > > epoll_wait(4, {}, 12, 0)                = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 933818440}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 934122715}) = 0
> > > > > > epoll_wait(4, {{EPOLLIN, {u32=1, u64=7284734596711710721}}}, 12, 35) =
> > > > > > 1
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 939059907}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 939326101}) = 0
> > > > > > recvmsg(8, {msg_name(16)={sa_family=AF_**INET, sin_port=htons(8999),
> > > > > > sin_addr=inet_addr("10.1.4.**124")}, msg_iov(1)=[{"\376\376\0\0\1\**
> > > > > > 2\"\377|\4\1\n\2|\4\1\n\2\0\n\**1\4|\0\0\0\0\0\0\0\0\0"..., 10000}],
> > > > > > msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 1472
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 939958334}) = 0
> > > > > > epoll_wait(4, {{EPOLLIN, {u32=1, u64=7284734596711710721}}, {EPOLLIN,
> > > > > > {u32=3, u64=8736411904714473475}}}, 12, 0) = 2
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 940609120}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 940881911}) = 0
> > > > > > recvmsg(8, {msg_name(16)={sa_family=AF_**INET, sin_port=htons(8999),
> > > > > > sin_addr=inet_addr("10.1.4.**124")}, msg_iov(1)=[{"\376\376\0\0\1\**
> > > > > > 2\"\377|\4\1\n\2|\4\1\n\2\0\n\**1\4|\0\0\0\0\0\0\0\0\0"..., 10000}],
> > > > > > msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 617
> > > > > > sendto(23, "\0", 1, MSG_NOSIGNAL, NULL, 0) = 1
> > > > > > recvmsg(12, {msg_name(16)={sa_family=AF_**INET, sin_port=htons(8999),
> > > > > > sin_addr=inet_addr("10.1.4.**133")}, msg_iov(1)=[{"\376\376\0\0\0\**
> > > > > > 0\"\377\205\4\1\n\262\0\0\0\**20=\0\0\262\0\0\0\0\0\0\0w\4\**1\n"...,
> > > > > > 10000}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 74
> > > > > > poll([{fd=8, events=POLLIN}], 1, 0)     = 1 ([{fd=8, revents=POLLIN}])
> > > > > > recvmsg(8, {msg_name(16)={sa_family=AF_**INET, sin_port=htons(8999),
> > > > > > sin_addr=inet_addr("10.1.4.**133")}, msg_iov(1)=[{"\376\376\0\0\5\**
> > > > > > 0\"\377\205\4\1\nw\4\1\n\2\0\**n\1\4w\0\0\0\0\0\0\0\0\0\0"...**,
> > > > > > 10000}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 42
> > > > > > sendmsg(11, {msg_name(16)={sa_family=AF_**INET, sin_port=htons(9000),
> > > > > > sin_addr=inet_addr("10.1.4.**121")}, msg_iov(1)=[{"\376\376\0\0\0\**
> > > > > > 0\"\377w\4\1\n\256\0\0\0\10=\**0\0\256\0\0\0\0\0\0\0w\4\1\n".**..,
> > > > > > 74}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 74
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 943922220}) = 0
> > > > > > poll([{fd=8, events=POLLIN}], 1, 0)     = 1 ([{fd=8, revents=POLLIN}])
> > > > > > recvmsg(8, {msg_name(16)={sa_family=AF_**INET, sin_port=htons(8999),
> > > > > > sin_addr=inet_addr("10.1.4.**133")}, msg_iov(1)=[{"\376\376\0\0\1\**
> > > > > > 2\"\377\205\4\1\n\2\205\4\1\n\**2\0\n\1\4\205\0\0\0\0\0\0\0\0\**0"...,
> > > > > > 10000}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 1472
> > > > > > poll([{fd=8, events=POLLIN}], 1, 0)     = 1 ([{fd=8, revents=POLLIN}])
> > > > > > recvmsg(8, {msg_name(16)={sa_family=AF_**INET, sin_port=htons(8999),
> > > > > > sin_addr=inet_addr("10.1.4.**133")}, msg_iov(1)=[{"\376\376\0\0\1\**
> > > > > > 2\"\377\205\4\1\n\2\205\4\1\n\**2\0\n\1\4\205\0\0\0\0\0\0\0\0\**0"...,
> > > > > > 10000}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 584
> > > > > > sendto(23, "\0", 1, MSG_NOSIGNAL, NULL, 0) = 1
> > > > > > poll([{fd=8, events=POLLIN}], 1, 0)     = 0 (Timeout)
> > > > > > poll([{fd=9, events=POLLIN}], 1, 0)     = 0 (Timeout)
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 946686072}) = 0
> > > > > > sendmsg(11, {msg_name(16)={sa_family=AF_**INET, sin_port=htons(9000),
> > > > > > sin_addr=inet_addr("10.1.4.**121")}, msg_iov(1)=[{"\376\376\0\0\0\**
> > > > > > 0\"\377w\4\1\n\262\0\0\0\21=\**0\0\262\0\0\0\0\0\0\0w\4\1\n".**..,
> > > > > > 74}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 74
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 952769552}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 952985732}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 953217383}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 953489345}) = 0
> > > > > > epoll_wait(4, {}, 12, 0)                = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 954019395}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 954314404}) = 0
> > > > > > epoll_wait(4, {{EPOLLIN, {u32=3, u64=8736411904714473475}}}, 12, 586)
> > > > > > = 1
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 954907616}) = 0
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 955138096}) = 0
> > > > > > recvmsg(12, {msg_name(16)={sa_family=AF_**INET, sin_port=htons(8999),
> > > > > > sin_addr=inet_addr("10.1.4.**133")}, msg_iov(1)=[{"\376\376\0\0\0\**
> > > > > > 0\"\377\205\4\1\n\262\0\0\0\**31=\0\0\262\0\0\0\0\0\0\0w\4\**1\n"...,
> > > > > > 10000}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 74
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 955849968}) = 0
> > > > > > poll([{fd=8, events=POLLIN}], 1, 0)     = 0 (Timeout)
> > > > > > poll([{fd=9, events=POLLIN}], 1, 0)     = 0 (Timeout)
> > > > > > clock_gettime(CLOCK_MONOTONIC, {57355, 956746383}) = 0
> > > > > > --- SIGBUS (Bus error) @ 0 (0) ---
> > > > > > Process 30095 detached
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > corosync-blackbox output
> > > > > >
> > > > > >
> > > > > > root@krusty:~# corosync-blackbox
> > > > > > Failed to initialize the cmap API. Error CS_ERR_LIBRARY
> > > > > > Failed to initialize the cmap API. Error CS_ERR_LIBRARY
> > > > > > Dumping the contents of /var/lib/corosync/fdata
> > > > > > [debug] shm size:8388608; real_size:8388608; rb->word_size:2097152
> > > > > > [debug] read total of: 8388620
> > > > > > Ringbuffer:
> > > > > > ->NORMAL
> > > > > > ->write_pt [673]
> > > > > > ->read_pt [0]
> > > > > > ->size [2097152 words]
> > > > > > =>free [8385912 bytes]
> > > > > > =>used [2692 bytes]
> > > > > > debug   May 07 15:26:52 handle_new_connection(476):2147483648: IPC
> > > > > > credentials authenticated (13438-14116-18)
> > > > > > debug   May 07 15:26:52 qb_ipcs_shm_connect(294):9: connecting to
> > > > > > client [14116]
> > > > > > debug   May 07 15:26:52 qb_rb_open_2(226):9: shm size:zd;
> > > > > > real_size:zd; rb->word_size:1048576
> > > > > > debug   May 07 15:26:52 qb_rb_open_2(226):9: shm size:zd;
> > > > > > real_size:zd; rb->word_size:1048576
> > > > > > debug   May 07 15:26:52 qb_rb_open_2(226):9: shm size:zd;
> > > > > > real_size:zd; rb->word_size:1048576
> > > > > > debug   May 07 15:26:52 cs_ipcs_connection_created(**269):8:
> > > > > > connection created
> > > > > > debug   May 07 15:26:52 cmap_lib_init_fn(306):9: lib_init_fn:
> > > > > > conn=0x7feb08b0ea00
> > > > > > debug   May 07 15:26:52 qb_ipcs_dispatch_connection_**request(723):9:
> > > > > > HUP conn (13438-14116-18)
> > > > > > debug   May 07 15:26:52 qb_ipcs_disconnect(565):9:
> > > > > > qb_ipcs_disconnect(13438-**14116-18) state:2
> > > > > > debug   May 07 15:26:52 _del(117):9: epoll_ctl(del): Bad file
> > > > > > descriptor (9)
> > > > > > debug   May 07 15:26:52 cs_ipcs_connection_closed(414)**:8:
> > > > > > cs_ipcs_connection_closed()
> > > > > > debug   May 07 15:26:52 cmap_lib_exit_fn(325):9: exit_fn for
> > > > > > conn=0x7feb08b0ea00
> > > > > > debug   May 07 15:26:52 cs_ipcs_connection_destroyed(**387):8:
> > > > > > cs_ipcs_connection_destroyed()
> > > > > > trace   May 07 15:26:52 qb_rb_close(279):9: ENTERING qb_rb_close()
> > > > > > trace   May 07 15:26:52 my_posix_sem_destroy(91):9: ENTERING
> > > > > > my_posix_sem_destroy()
> > > > > > debug   May 07 15:26:52 qb_rb_close(290):9: Free'ing ringbuffer:
> > > > > > /dev/shm/qb-cmap-response-**13438-14116-18-header
> > > > > > trace   May 07 15:26:52 qb_rb_close(279):9: ENTERING qb_rb_close()
> > > > > > trace   May 07 15:26:52 my_posix_sem_destroy(91):9: ENTERING
> > > > > > my_posix_sem_destroy()
> > > > > > debug   May 07 15:26:52 qb_rb_close(290):9: Free'ing ringbuffer:
> > > > > > /dev/shm/qb-cmap-event-13438-**14116-18-header
> > > > > > trace   May 07 15:26:52 qb_rb_close(279):9: ENTERING qb_rb_close()
> > > > > > trace   May 07 15:26:52 my_posix_sem_destroy(91):9: ENTERING
> > > > > > my_posix_sem_destroy()
> > > > > > debug   May 07 15:26:52 qb_rb_close(290):9: Free'ing ringbuffer:
> > > > > > /dev/shm/qb-cmap-request-**13438-14116-18-header
> > > > > > debug   May 07 15:26:52 handle_new_connection(476):2147483648: IPC
> > > > > > credentials authenticated (13438-14118-18)
> > > > > > debug   May 07 15:26:52 qb_ipcs_shm_connect(294):9: connecting to
> > > > > > client [14118]
> > > > > > debug   May 07 15:26:52 qb_rb_open_2(226):9: shm size:zd;
> > > > > > real_size:zd; rb->word_size:1048576
> > > > > > debug   May 07 15:26:52 qb_rb_open_2(226):9: shm size:zd;
> > > > > > real_size:zd; rb->word_size:1048576
> > > > > > debug   May 07 15:26:52 qb_rb_open_2(226):9: shm size:zd;
> > > > > > real_size:zd; rb->word_size:1048576
> > > > > > debug   May 07 15:26:52 cs_ipcs_connection_created(**269):8:
> > > > > > connection created
> > > > > > debug   May 07 15:26:52 cmap_lib_init_fn(306):9: lib_init_fn:
> > > > > > conn=0x7feb08b0ea00
> > > > > > ERROR: qb_rb_chunk_read failed: Connection timed out
> > > > > > [trace] ENTERING qb_rb_close()
> > > > > > [debug] Free'ing ringbuffer: /dev/shm/qb-create_from_file-**header
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > core dump
> > > > > >
> > > > > > Reading symbols from /usr/sbin/corosync...done.
> > > > > > [New LWP 30095]
> > > > > > [New LWP 30096]
> > > > > >
> > > > > > warning: Can't read pathname for load map: Input/output error.
> > > > > > [Thread debugging using libthread_db enabled]
> > > > > > Using host libthread_db library "/lib/x86_64-linux-gnu/**
> > > > > > libthread_db.so.1".
> > > > > > Core was generated by `corosync'.
> > > > > > Program terminated with signal 7, Bus error.
> > > > > > #0  0x00007f418e4ea823 in qb_rb_chunk_alloc (rb=0x7f418f9c0d20,
> > > > > > len=<optimized out>) at ringbuffer.c:456
> > > > > > 456             rb->shared_data[write_pt] = 0;
> > > > > >
> > > > > > (gdb) thread apply all bt
> > > > > >
> > > > > > Thread 2 (Thread 0x7f418bf6f700 (LWP 30096)):
> > > > > > #0  0x00007f418e2cdfd0 in sem_wait () from /lib/x86_64-linux-gnu/**
> > > > > > libpthread.so.0
> > > > > > #1  0x00007f418e4f45f3 in qb_logt_worker_thread (data=<optimized out>)
> > > > > > at log_thread.c:71
> > > > > > #2  0x00007f418e2c7e9a in start_thread () from /lib/x86_64-linux-gnu/*
> > > > > > *libpthread.so.0
> > > > > > #3  0x00007f418dff3cbd in clone () from /lib/x86_64-linux-gnu/libc.so.
> > > > > > **6
> > > > > > #4  0x0000000000000000 in ?? ()
> > > > > >
> > > > > > Thread 1 (Thread 0x7f418ed96700 (LWP 30095)):
> > > > > > #0  0x00007f418e4ea823 in qb_rb_chunk_alloc (rb=0x7f418f9c0d20,
> > > > > > len=<optimized out>) at ringbuffer.c:456
> > > > > > #1  0x00007f418e4f4bb9 in _blackbox_vlogger (target=<optimized out>,
> > > > > > cs=0x7f418fa3e4d8, timestamp=1368027286,
> > > > > >    ap=0x7fff3408ebf8) at log_blackbox.c:74
> > > > > > #2  0x00007f418e4f2dea in qb_log_real_va_ (cs=0x7f418fa3e4d8,
> > > > > > ap=0x7fff3408ee88) at log.c:184
> > > > > > #3  0x00007f418edbdf00 in _logsys_log_printf (level=<optimized out>,
> > > > > > subsys=<optimized out>,
> > > > > >    function_name=<optimized out>, file_name=<optimized out>,
> > > > > > file_line=<optimized out>, format=<optimized out>)
> > > > > >    at main.c:898
> > > > > > #4  0x00007f418e95f7bd in messages_free (token_aru=<optimized out>,
> > > > > > instance=0x7f418b732010) at totemsrp.c:2462
> > > > > > #5  message_handler_orf_token (instance=0x7f418b732010, msg=<optimized
> > > > > > out>, endian_conversion_needed=<**optimized out>,
> > > > > >    msg_len=<optimized out>) at totemsrp.c:3611
> > > > > > #6  0x00007f418e9617f7 in message_handler_orf_token
> > > > > > (instance=<optimized out>, msg=<optimized out>,
> > > > > >    msg_len=<optimized out>, endian_conversion_needed=<**optimized
> > > > > > out>) at totemsrp.c:3532
> > > > > > #7  0x00007f418e95bcc1 in rrp_deliver_fn (context=0x7f418f9ce850,
> > > > > > msg=0x7f418fa0f868, msg_len=70) at totemrrp.c:1783
> > > > > > #8  0x00007f418e956ad2 in net_deliver_fn (fd=<optimized out>,
> > > > > > revents=<optimized out>, data=0x7f418fa0f800)
> > > > > >    at totemudp.c:521
> > > > > > #9  0x00007f418e4ec05f in _poll_dispatch_and_take_back_
> > > > > > (item=<optimized out>, p=<optimized out>) at loop_poll.c:98
> > > > > > #10 0x00007f418e4ebbd7 in qb_loop_run_level (level=0x7f418f9c7268) at
> > > > > > loop.c:43
> > > > > > #11 qb_loop_run (lp=<optimized out>) at loop.c:204
> > > > > > #12 0x00007f418edacbe5 in main (argc=<optimized out>, argv=<optimized
> > > > > > out>, envp=<optimized out>) at main.c:1309
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > syslog
> > > > > >
> > > > > >
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo
> > > > > > message from cluster node 167838851
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo
> > > > > > message[167838851]: votes: 1, expected: 17 flags: 1
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] flags: quorate: Yes
> > > > > > Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No
> > > > > > QdeviceCastVote: No QdeviceMasterWins: No
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] total_votes=8,
> > > > > > expected_votes=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838839
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838841
> > > > > > state=2, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838842
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838844
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838846
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838849
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838850
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838851
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838853
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] quorum lost,
> > > > > > blocking activity
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo
> > > > > > message from cluster node 167838851
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo
> > > > > > message[0]: votes: 0, expected: 0 flags: 0
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo
> > > > > > message from cluster node 167838853
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo
> > > > > > message[167838853]: votes: 1, expected: 17 flags: 1
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] flags: quorate: Yes
> > > > > > Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No
> > > > > > QdeviceCastVote: No QdeviceMasterWins: No
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] total_votes=8,
> > > > > > expected_votes=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838839
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838841
> > > > > > state=2, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838842
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838844
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838846
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838849
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838850
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838851
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838853
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo
> > > > > > message from cluster node 167838853
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo
> > > > > > message[0]: votes: 0, expected: 0 flags: 0
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo
> > > > > > message from cluster node 167838839
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo
> > > > > > message[167838839]: votes: 1, expected: 17 flags: 1
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] flags: quorate: Yes
> > > > > > Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No
> > > > > > QdeviceCastVote: No QdeviceMasterWins: No
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] total_votes=8,
> > > > > > expected_votes=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838839
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838841
> > > > > > state=2, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838842
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838844
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838846
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838849
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838850
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838851
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838853
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo
> > > > > > message from cluster node 167838839
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo
> > > > > > message[0]: votes: 0, expected: 0 flags: 0
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo
> > > > > > message from cluster node 167838842
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo
> > > > > > message[167838842]: votes: 1, expected: 17 flags: 1
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] flags: quorate: Yes
> > > > > > Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No
> > > > > > QdeviceCastVote: No QdeviceMasterWins: No
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] total_votes=8,
> > > > > > expected_votes=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838839
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838841
> > > > > > state=2, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838842
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838844
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838846
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838849
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838850
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838851
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838853
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo
> > > > > > message from cluster node 167838842
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo
> > > > > > message[0]: votes: 0, expected: 0 flags: 0
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo
> > > > > > message from cluster node 167838844
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo
> > > > > > message[167838844]: votes: 1, expected: 17 flags: 1
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] flags: quorate: Yes
> > > > > > Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No
> > > > > > QdeviceCastVote: No QdeviceMasterWins: No
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] total_votes=8,
> > > > > > expected_votes=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838839
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838841
> > > > > > state=2, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838842
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838844
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838846
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838849
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838850
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838851
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838853
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo
> > > > > > message from cluster node 167838844
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo
> > > > > > message[0]: votes: 0, expected: 0 flags: 0
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] got nodeinfo
> > > > > > message from cluster node 167838846
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] nodeinfo
> > > > > > message[167838846]: votes: 1, expected: 17 flags: 1
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] flags: quorate: Yes
> > > > > > Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No
> > > > > > QdeviceCastVote: No QdeviceMasterWins: No
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] total_votes=8,
> > > > > > expected_votes=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838839
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838841
> > > > > > state=2, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838842
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838844
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838846
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty corosync[30095]:   [QUORUM] node 167838849
> > > > > > state=1, votes=1, expected=17
> > > > > > May  8 10:26:23 krusty rsyslogd-2177: imuxsock begins to drop messages
> > > > > > from pid 30095 due to rate-limiting
> > > > > > May  8 10:26:23 krusty crmd[30689]:   notice:
> > > > > > pcmk_quorum_notification: Membership 47416: quorum lost (8)
> > > > > > May  8 10:26:24 krusty crmd[30689]:   notice:
> > > > > > pcmk_quorum_notification: Membership 47420: quorum acquired (9)
> > > > > > May  8 10:26:25 krusty crmd[30689]:   notice: corosync_node_name:
> > > > > > Unable to get node name for nodeid 167838841
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:   notice: unpack_config: On
> > > > > > loss of CCM Quorum: Ignore
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=quimby - this is rarely intended
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=todd - this is rarely intended
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=rod - this is rarely intended
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=maude - this is rarely intended
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=krusty - this is rarely intended
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=itchy - this is rarely intended
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=sideshowbob - this is rarely
> > > > > > intended
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=willie - this is rarely intended
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=ned - this is rarely intended
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=moe - this is rarely intended
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=nelson - this is rarely intended
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=scratchy - this is rarely intended
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=fattony - this is rarely intended
> > > > > > May  8 10:27:37 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=barney - this is rarely intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:   notice: unpack_config: On
> > > > > > loss of CCM Quorum: Ignore
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=quimby - this is rarely intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=todd - this is rarely intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=rod - this is rarely intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=maude - this is rarely intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=krusty - this is rarely intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=itchy - this is rarely intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=sideshowbob - this is rarely
> > > > > > intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=willie - this is rarely intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=ned - this is rarely intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=moe - this is rarely intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=nelson - this is rarely intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=scratchy - this is rarely intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=fattony - this is rarely intended
> > > > > > May  8 10:32:47 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=barney - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:   notice: unpack_config: On
> > > > > > loss of CCM Quorum: Ignore
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=quimby - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=todd - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=rod - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=maude - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=krusty - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=itchy - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=sideshowbob - this is rarely
> > > > > > intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=willie - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=ned - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=moe - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=nelson - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=scratchy - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=fattony - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning: create_node:
> > > > > > Detected multiple node entries with uname=barney - this is rarely intended
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning:
> > > > > > unpack_rsc_location: No resource (con=cli-standby-res_IPaddr2_**NXTProduction,
> > > > > > rsc=res_IPaddr2_NXTProduction)
> > > > > > May  8 10:34:23 krusty stonith-ng[30687]:  warning:
> > > > > > unpack_rsc_location: No resource (con=cli-prefer-res_IPaddr2_**NXTProduction,
> > > > > > rsc=res_IPaddr2_NXTProduction)
> > > > > > May  8 10:34:46 krusty pacemakerd[30683]:    error:
> > > > > > cfg_connection_destroy: Connection destroyed
> > > > > > May  8 10:34:46 krusty pacemakerd[30683]:   notice:
> > > > > > pcmk_shutdown_worker: Shuting down Pacemaker
> > > > > > May  8 10:34:46 krusty pacemakerd[30683]:   notice: stop_child:
> > > > > > Stopping crmd: Sent -15 to process 30689
> > > > > > May  8 10:34:46 krusty crmd[30689]:   notice: crm_shutdown: Requesting
> > > > > > shutdown, upper limit is 1200000ms
> > > > > > May  8 10:34:46 krusty pacemakerd[30683]:    error:
> > > > > > cpg_connection_destroy: Connection destroyed
> > > > > > May  8 10:34:46 krusty stonith-ng[30687]:    error: pcmk_cpg_dispatch:
> > > > > > Connection to the CPG API failed: 2
> > > > > > May  8 10:34:46 krusty stonith-ng[30687]:    error:
> > > > > > stonith_peer_ais_destroy: AIS connection terminated
> > > > > > May  8 10:34:46 krusty crmd[30689]:    error: crm_ipc_read: Connection
> > > > > > to stonith-ng failed
> > > > > > May  8 10:34:46 krusty crmd[30689]:    error: mainloop_gio_callback:
> > > > > > Connection to stonith-ng[0x1cce4e0] closed (I/O condition=17)
> > > > > > May  8 10:34:46 krusty crmd[30689]:     crit:
> > > > > > tengine_stonith_connection_**destroy: Fencing daemon connection failed
> > > > > > May  8 10:34:46 krusty attrd[30688]:    error: pcmk_cpg_dispatch:
> > > > > > Connection to the CPG API failed: 2
> > > > > > May  8 10:34:46 krusty attrd[30688]:     crit: attrd_ais_destroy: Lost
> > > > > > connection to Corosync service!
> > > > > > May  8 10:34:46 krusty attrd[30688]:   notice: main: Exiting...
> > > > > > May  8 10:34:46 krusty attrd[30688]:   notice: main: Disconnecting
> > > > > > client 0x22e9eb0, pid=30689...
> > > > > > May  8 10:34:46 krusty attrd[30688]:    error:
> > > > > > attrd_cib_connection_destroy: Connection to the CIB terminated...
> > > > > > May  8 10:34:46 krusty cib[30686]:    error: pcmk_cpg_dispatch:
> > > > > > Connection to the CPG API failed: 2
> > > > > > May  8 10:34:46 krusty cib[30686]:    error: cib_ais_destroy: Corosync
> > > > > > connection lost!  Exiting.
> > > > > > May  8 10:34:47 krusty crmd[30689]:    error: te_connect_stonith:
> > > > > > Sign-in failed: triggered a retry
> > > > > > May  8 10:34:47 krusty crmd[30689]:    error: crm_ipc_read: Connection
> > > > > > to cib_shm failed
> > > > > > May  8 10:34:47 krusty crmd[30689]:    error: mainloop_gio_callback:
> > > > > > Connection to cib_shm[0x1baf020] closed (I/O condition=17)
> > > > > > May  8 10:34:47 krusty crmd[30689]:    error:
> > > > > > crmd_cib_connection_destroy: Connection to the CIB terminated...
> > > > > > May  8 10:34:47 krusty crmd[30689]:    error: crmd_quorum_destroy:
> > > > > > connection terminated
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ______________________________**_________________
> > > > > > discuss mailing list
> > > > > > discuss@xxxxxxxxxxxx
> > > > > > http://lists.corosync.org/**mailman/listinfo/discuss<http://lists.corosync.org/mailman/listinfo/discuss>
> > >  ______________________________**_________________
> > discuss mailing list
> > discuss@xxxxxxxxxxxx
> > http://lists.corosync.org/**mailman/listinfo/discuss<http://lists.corosync.org/mailman/listinfo/discuss>

> _______________________________________________
> discuss mailing list
> discuss@xxxxxxxxxxxx
> http://lists.corosync.org/mailman/listinfo/discuss

_______________________________________________
discuss mailing list
discuss@xxxxxxxxxxxx
http://lists.corosync.org/mailman/listinfo/discuss