On Fri, Jun 22, 2018 at 8:51 AM David Zafman <dzafman@xxxxxxxxxx> wrote: > > > These are the Centos run stack traces generated with docker using the > same runs as I had been doing for > https://tracker.ceph.com/issues/23492. The crashes under Ubuntu always > involved either a decode() crash, decode() exception or assert(st_size > != 0). > > Is this a clue, or is it just a Centos anomaly? BTW, all other threads > in these cores look fine! If the other threads look fine, that makes it sound a lot like this is a clue and there's some memory corruption happening to the stack. Which makes sense since you're seeing OSDMaps getting overwritten with log data here, right? -Greg > > 2 failures in > http://pulpito.ceph.com/dzafman-2018-06-21_15:29:45-rados:standalone-wip-zafman-testing2-distro-basic-smithi/ > > > # sudo ./ceph-debug-docker.sh > wip-zafman-testing2:d12ea8b6b641958cdfcf609d2fad8947a21965cf centos:7 > > # gdb /usr/bin/ceph-osd > /ceph/teuthology-archive/dzafman-2018-06-21_15:29:45-rados:standalone-wip-zafman-testing2-distro-basic-smithi/2687446/remote/smithi168/coredump/1529622699.99940.core > ... > warning: .dynamic section for "/lib64/libudev.so.1" is not at the > expected address (wrong library or version mismatch?) > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib64/libthread_db.so.1". > Core was generated by `ceph-osd -i 3 > --fsid=f2babf17-f782-48ce-9563-54ab3eb8dc70 --auth-supported=none'. > Program terminated with signal 6, Aborted. > #0 0x00007f9d992b659b in raise () from /lib64/libpthread.so.0 > Missing separate debuginfos, use: debuginfo-install > bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.170-4.el7.x86_64 > elfutils-libs-0.170-4.el7.x86_64 fuse-libs-2.9.2-10.el7.x86_64 > glibc-2.17-222.el7.x86_64 gperftools-libs-2.6.1-1.el7.x86_64 > keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-19.el7.x86_64 > leveldb-1.12.0-11.el7.x86_64 libaio-0.3.109-13.el7.x86_64 > libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-52.el7.x86_64 > libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-12.el7_5.x86_64 > libgcc-4.8.5-28.el7_5.1.x86_64 libibverbs-15-7.el7_5.x86_64 > libnl3-3.2.28-4.el7.x86_64 liboath-2.4.1-9.el7.x86_64 > libselinux-2.5-12.el7.x86_64 libstdc++-4.8.5-28.el7_5.1.x86_64 > libuuid-2.23.2-52.el7.x86_64 lttng-ust-2.4.1-4.el7.x86_64 > lz4-1.7.5-2.el7.x86_64 nspr-4.19.0-1.el7_5.x86_64 > nss-3.36.0-5.el7_5.x86_64 nss-softokn-3.36.0-5.el7_5.x86_64 > nss-softokn-freebl-3.36.0-5.el7_5.x86_64 nss-util-3.36.0-1.el7_5.x86_64 > openssl-libs-1.0.2k-12.el7.x86_64 pcre-8.32-17.el7.x86_64 > snappy-1.1.0-3.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 > systemd-libs-219-57.el7.x86_64 userspace-rcu-0.7.16-1.el7.x86_64 > xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64 > (gdb) thread > [Current thread is 1 (Thread 0x7f9d7671f700 (LWP 100040))] > (gdb) bt > #0 0x00007f9d992b659b in raise () from /lib64/libpthread.so.0 > #1 0x0000558705d5f521 in reraise_fatal (signum=6) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/global/signal_handler.cc:74 > #2 handle_fatal_signal (signum=6) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/global/signal_handler.cc:138 > #3 <signal handler called> > #4 0x00007f9d982d6277 in raise () from /lib64/libc.so.6 > #5 0x00007f9d982d7968 in abort () from /lib64/libc.so.6 > #6 0x00007f9d98be5ac5 in __cxa_vec_dtor () from /lib64/libstdc++.so.6 > #7 0x00007f9d98be3a63 in ?? () from /lib64/libstdc++.so.6 > #8 0x00007f9d76719a20 in ?? () > #9 0x0000000000000ae0 in ?? () > #10 0x00007f9d9b1fe5c6 in (anonymous namespace)::do_memalign(unsigned > long, unsigned long) () from /lib64/libtcmalloc.so.4 > #11 0x00007f9d9b21d010 in tc_posix_memalign () from /lib64/libtcmalloc.so.4 > #12 0x00007f9d9b20aacc in tcmalloc::PageHeap::Carve(tcmalloc::Span*, > unsigned long) () from /lib64/libtcmalloc.so.4 > #13 0x00007f9d9b20b591 in tcmalloc::PageHeap::New(unsigned long) () from > /lib64/libtcmalloc.so.4 > #14 0x00007f9d9b20a230 in tcmalloc::CentralFreeList::Populate() () from > /lib64/libtcmalloc.so.4 > #15 0x00007f9d767199d0 in ?? () > #16 0x0000000000000000 in ?? () > > # gdb /usr/bin/ceph-osd > /ceph/teuthology-archive/dzafman-2018-06-21_15:29:45-rados:standalone-wip-zafman-testing2-distro-basic-smithi/2687443/remote/smithi111/coredump/1529622850.110914.core > > Thread 1 (Thread 0x7fe4bf8e2700 (LWP 111012)): > #0 0x00007fe4e247959b in raise () from /lib64/libpthread.so.0 > #1 0x0000562bfbd41521 in reraise_fatal (signum=6) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/global/signal_handler.cc:74 > #2 handle_fatal_signal (signum=6) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/global/signal_handler.cc:138 > #3 <signal handler called> > #4 0x00007fe4e1499277 in raise () from /lib64/libc.so.6 > #5 0x00007fe4e149a968 in abort () from /lib64/libc.so.6 > #6 0x00007fe4e1da8ac5 in __cxa_vec_dtor () from /lib64/libstdc++.so.6 > #7 0x00007fe4e1da6a63 in ?? () from /lib64/libstdc++.so.6 > #8 0x00007fe4bf8dca20 in ?? () > #9 0x0000000000000ae0 in ?? () > #10 0x00007fe4e43c15c6 in (anonymous namespace)::do_memalign(unsigned > long, unsigned long) () from /lib64/libtcmalloc.so.4 > #11 0x00007fe4e43e0010 in tc_posix_memalign () from /lib64/libtcmalloc.so.4 > #12 0x00007fe4e58cbc23 in raw (mempool=10, l=2710, c=<optimized out>, > this=0x8) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/include/buffer_raw.h:44 > #13 raw_combined (mempool=10, align=2710, l=2710, dataptr=<optimized > out>, this=0x8) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/buffer.cc:181 > #14 create (mempool=10, align=2710, len=2710) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/buffer.cc:214 > #15 ceph::buffer::create_aligned_in_mempool (len=2710, align=2710, > mempool=10) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/buffer.cc:709 > #16 0x0cb4eb4a50de6700 in ?? () > #17 0x0000562bfc425940 in ?? () > #18 0x0000562bfbb5a030 in FileStore::read (this=<optimized out>, ch=..., > oid=..., offset=<optimized out>, len=<optimized out>, bl=..., > op_flags=4222983760) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/os/filestore/FileStore.cc:3382 > #19 0x0000000000000a96 in ?? () > #20 0x0000562bfbb59e50 in ?? () at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/os/filestore/FileStore.cc:1909 > #21 0x0000562bfb8053ed in OSDService::_get_map_bl (this=0x3a, > e=1356752640, bl=...) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:1364 > #22 0x0000562bfb812866 in OSDService::try_get_map (this=0x562bfdca6360, > this@entry=0x562bfdb433e0, epoch=<optimized out>, epoch@entry=58) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:1480 > #23 0x0000562bfb81b2dd in OSD::advance_pg > (this=this@entry=0x562bfdb42000, osd_epoch=<optimized out>, > pg=pg@entry=0x562bfdc34000, handle=..., rctx=rctx@entry=0x7fe4bf8dcf90) > at /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:7747 > #24 0x0000562bfb81ba81 in OSD::dequeue_peering_evt (this=0x562bfdb42000, > sdata=<optimized out>, pg=0x562bfdc34000, evt=std::shared_ptr (count 2, > weak 0) 0x562bfdc40f90, handle=...) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:8877 > #25 0x0000562bfba741a0 in PGPeeringItem::run (this=<optimized out>, > osd=<optimized out>, sdata=<optimized out>, pg=..., handle=...) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OpQueueItem.cc:34 > #26 0x0000562bfb824532 in run (handle=..., pg=..., sdata=<optimized > out>, osd=<optimized out>, this=0x7fe4bf8dd140) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OpQueueItem.h:134 > #27 OSD::ShardedOpWQ::_process (this=0x562bfdb43048, > thread_index=<optimized out>, hb=<optimized out>) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:9849 > #28 0x00007fe4e5923f63 in ShardedThreadPool::shardedthreadpool_worker > (this=0x562bfdb42930, thread_index=<optimized out>) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/WorkQueue.cc:339 > #29 0x00007fe4e5924b50 in ShardedThreadPool::WorkThreadSharded::entry > (this=<optimized out>) at > /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/WorkQueue.h:690 > #30 0x00007fe4e2471e25 in start_thread () from /lib64/libpthread.so.0 > #31 0x00007fe4e1561bad in clone () from /lib64/libc.so.6 > > David > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
