These are the Centos run stack traces generated with docker using the
same runs as I had been doing for
https://tracker.ceph.com/issues/23492. ; The crashes under Ubuntu always
involved either a decode() crash, decode() exception or assert(st_size
!= 0).
Is this a clue, or is it just a Centos anomaly? BTW, all other threads
in these cores look fine!
2 failures in
http://pulpito.ceph.com/dzafman-2018-06-21_15:29:45-rados:standalone-wip-zafman-testing2-distro-basic-smithi/
# sudo ./ceph-debug-docker.sh
wip-zafman-testing2:d12ea8b6b641958cdfcf609d2fad8947a21965cf centos:7
# gdb /usr/bin/ceph-osd
/ceph/teuthology-archive/dzafman-2018-06-21_15:29:45-rados:standalone-wip-zafman-testing2-distro-basic-smithi/2687446/remote/smithi168/coredump/1529622699.99940.core
...
warning: .dynamic section for "/lib64/libudev.so.1" is not at the
expected address (wrong library or version mismatch?)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `ceph-osd -i 3
--fsid=f2babf17-f782-48ce-9563-54ab3eb8dc70 --auth-supported=none'.
Program terminated with signal 6, Aborted.
#0 0x00007f9d992b659b in raise () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install
bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.170-4.el7.x86_64
elfutils-libs-0.170-4.el7.x86_64 fuse-libs-2.9.2-10.el7.x86_64
glibc-2.17-222.el7.x86_64 gperftools-libs-2.6.1-1.el7.x86_64
keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-19.el7.x86_64
leveldb-1.12.0-11.el7.x86_64 libaio-0.3.109-13.el7.x86_64
libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-52.el7.x86_64
libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-12.el7_5.x86_64
libgcc-4.8.5-28.el7_5.1.x86_64 libibverbs-15-7.el7_5.x86_64
libnl3-3.2.28-4.el7.x86_64 liboath-2.4.1-9.el7.x86_64
libselinux-2.5-12.el7.x86_64 libstdc++-4.8.5-28.el7_5.1.x86_64
libuuid-2.23.2-52.el7.x86_64 lttng-ust-2.4.1-4.el7.x86_64
lz4-1.7.5-2.el7.x86_64 nspr-4.19.0-1.el7_5.x86_64
nss-3.36.0-5.el7_5.x86_64 nss-softokn-3.36.0-5.el7_5.x86_64
nss-softokn-freebl-3.36.0-5.el7_5.x86_64 nss-util-3.36.0-1.el7_5.x86_64
openssl-libs-1.0.2k-12.el7.x86_64 pcre-8.32-17.el7.x86_64
snappy-1.1.0-3.el7.x86_64 sqlite-3.7.17-8.el7.x86_64
systemd-libs-219-57.el7.x86_64 userspace-rcu-0.7.16-1.el7.x86_64
xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) thread
[Current thread is 1 (Thread 0x7f9d7671f700 (LWP 100040))]
(gdb) bt
#0 0x00007f9d992b659b in raise () from /lib64/libpthread.so.0
#1 0x0000558705d5f521 in reraise_fatal (signum=6) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/global/signal_handler.cc:74
#2 handle_fatal_signal (signum=6) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/global/signal_handler.cc:138
#3 <signal handler called>
#4 0x00007f9d982d6277 in raise () from /lib64/libc.so.6
#5 0x00007f9d982d7968 in abort () from /lib64/libc.so.6
#6 0x00007f9d98be5ac5 in __cxa_vec_dtor () from /lib64/libstdc++.so.6
#7 0x00007f9d98be3a63 in ?? () from /lib64/libstdc++.so.6
#8 0x00007f9d76719a20 in ?? ()
#9 0x0000000000000ae0 in ?? ()
#10 0x00007f9d9b1fe5c6 in (anonymous namespace)::do_memalign(unsigned
long, unsigned long) () from /lib64/libtcmalloc.so.4
#11 0x00007f9d9b21d010 in tc_posix_memalign () from /lib64/libtcmalloc.so.4
#12 0x00007f9d9b20aacc in tcmalloc::PageHeap::Carve(tcmalloc::Span*,
unsigned long) () from /lib64/libtcmalloc.so.4
#13 0x00007f9d9b20b591 in tcmalloc::PageHeap::New(unsigned long) () from
/lib64/libtcmalloc.so.4
#14 0x00007f9d9b20a230 in tcmalloc::CentralFreeList::Populate() () from
/lib64/libtcmalloc.so.4
#15 0x00007f9d767199d0 in ?? ()
#16 0x0000000000000000 in ?? ()
# gdb /usr/bin/ceph-osd
/ceph/teuthology-archive/dzafman-2018-06-21_15:29:45-rados:standalone-wip-zafman-testing2-distro-basic-smithi/2687443/remote/smithi111/coredump/1529622850.110914.core
Thread 1 (Thread 0x7fe4bf8e2700 (LWP 111012)):
#0 0x00007fe4e247959b in raise () from /lib64/libpthread.so.0
#1 0x0000562bfbd41521 in reraise_fatal (signum=6) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/global/signal_handler.cc:74
#2 handle_fatal_signal (signum=6) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/global/signal_handler.cc:138
#3 <signal handler called>
#4 0x00007fe4e1499277 in raise () from /lib64/libc.so.6
#5 0x00007fe4e149a968 in abort () from /lib64/libc.so.6
#6 0x00007fe4e1da8ac5 in __cxa_vec_dtor () from /lib64/libstdc++.so.6
#7 0x00007fe4e1da6a63 in ?? () from /lib64/libstdc++.so.6
#8 0x00007fe4bf8dca20 in ?? ()
#9 0x0000000000000ae0 in ?? ()
#10 0x00007fe4e43c15c6 in (anonymous namespace)::do_memalign(unsigned
long, unsigned long) () from /lib64/libtcmalloc.so.4
#11 0x00007fe4e43e0010 in tc_posix_memalign () from /lib64/libtcmalloc.so.4
#12 0x00007fe4e58cbc23 in raw (mempool=10, l=2710, c=<optimized out>,
this=0x8) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/include/buffer_raw.h:44
#13 raw_combined (mempool=10, align=2710, l=2710, dataptr=<optimized
out>, this=0x8) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/buffer.cc:181
#14 create (mempool=10, align=2710, len=2710) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/buffer.cc:214
#15 ceph::buffer::create_aligned_in_mempool (len=2710, align=2710,
mempool=10) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/buffer.cc:709
#16 0x0cb4eb4a50de6700 in ?? ()
#17 0x0000562bfc425940 in ?? ()
#18 0x0000562bfbb5a030 in FileStore::read (this=<optimized out>, ch=...,
oid=..., offset=<optimized out>, len=<optimized out>, bl=...,
op_flags=4222983760) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/os/filestore/FileStore.cc:3382
#19 0x0000000000000a96 in ?? ()
#20 0x0000562bfbb59e50 in ?? () at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/os/filestore/FileStore.cc:1909
#21 0x0000562bfb8053ed in OSDService::_get_map_bl (this=0x3a,
e=1356752640, bl=...) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:1364
#22 0x0000562bfb812866 in OSDService::try_get_map (this=0x562bfdca6360,
this@entry=0x562bfdb433e0, epoch=<optimized out>, epoch@entry=58) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:1480
#23 0x0000562bfb81b2dd in OSD::advance_pg
(this=this@entry=0x562bfdb42000, osd_epoch=<optimized out>,
pg=pg@entry=0x562bfdc34000, handle=..., rctx=rctx@entry=0x7fe4bf8dcf90)
at /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:7747
#24 0x0000562bfb81ba81 in OSD::dequeue_peering_evt (this=0x562bfdb42000,
sdata=<optimized out>, pg=0x562bfdc34000, evt=std::shared_ptr (count 2,
weak 0) 0x562bfdc40f90, handle=...) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:8877
#25 0x0000562bfba741a0 in PGPeeringItem::run (this=<optimized out>,
osd=<optimized out>, sdata=<optimized out>, pg=..., handle=...) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OpQueueItem.cc:34
#26 0x0000562bfb824532 in run (handle=..., pg=..., sdata=<optimized
out>, osd=<optimized out>, this=0x7fe4bf8dd140) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OpQueueItem.h:134
#27 OSD::ShardedOpWQ::_process (this=0x562bfdb43048,
thread_index=<optimized out>, hb=<optimized out>) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:9849
#28 0x00007fe4e5923f63 in ShardedThreadPool::shardedthreadpool_worker
(this=0x562bfdb42930, thread_index=<optimized out>) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/WorkQueue.cc:339
#29 0x00007fe4e5924b50 in ShardedThreadPool::WorkThreadSharded::entry
(this=<optimized out>) at
/usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/WorkQueue.h:690
#30 0x00007fe4e2471e25 in start_thread () from /lib64/libpthread.so.0
#31 0x00007fe4e1561bad in clone () from /lib64/libc.so.6
David
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
