The first stack looks to be so badly smashed that gdb can't unwind it. The second stack also appears to be badly corrupted. (gdb) f #22 0x0000562bfb812866 in OSDService::try_get_map (this=0x562bfdca6360, this@entry=0x562bfdb433e0, epoch=<optimized out>, epoch@entry=58) at /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:1480 1480 map->decode(bl); (gdb) p &bl $13 = (ceph::bufferlist *) 0x7fe4bf8dccf0 (gdb) p $rsp $15 = (void *) 0x7fe4bf8dccc0 So bl is a bufferlist variable on the stack we are passing to decode. (gdb) down #21 0x0000562bfb8053ed in OSDService::_get_map_bl (this=0x3a, e=1356752640, bl=...) at /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:1364 1364 _add_map_bl(e, bl); (gdb) p bl $16 = (ceph::bufferlist &) @0xcb4eb4a50de6700: <error reading variable> (gdb) p $rsp $18 = (void *) 0x7fe4bf8dcc20 (gdb) x/5x 0x7fe4bf8dcc20 0x7fe4bf8dcc20: 0x0000000000001002 0x0cb4eb4a50de6700 0x7fe4bf8dcc30: 0x0000562bfdc6fc38 0x00007fe4bf8dcc70 0x7fe4bf8dcc40: 0x0000562bfdc7b200 So it looks like the stack has been overwritten with some data and the stack can not be trusted. On Sat, Jun 23, 2018 at 6:35 AM, Gregory Farnum <gfarnum@xxxxxxxxxx> wrote: > On Fri, Jun 22, 2018 at 8:51 AM David Zafman <dzafman@xxxxxxxxxx> wrote: >> >> >> These are the Centos run stack traces generated with docker using the >> same runs as I had been doing for >> https://tracker.ceph.com/issues/23492. The crashes under Ubuntu always >> involved either a decode() crash, decode() exception or assert(st_size >> != 0). >> >> Is this a clue, or is it just a Centos anomaly? BTW, all other threads >> in these cores look fine! > > If the other threads look fine, that makes it sound a lot like this is > a clue and there's some memory corruption happening to the stack. > Which makes sense since you're seeing OSDMaps getting overwritten with > log data here, right? > -Greg > >> >> 2 failures in >> http://pulpito.ceph.com/dzafman-2018-06-21_15:29:45-rados:standalone-wip-zafman-testing2-distro-basic-smithi/ >> >> >> # sudo ./ceph-debug-docker.sh >> wip-zafman-testing2:d12ea8b6b641958cdfcf609d2fad8947a21965cf centos:7 >> >> # gdb /usr/bin/ceph-osd >> /ceph/teuthology-archive/dzafman-2018-06-21_15:29:45-rados:standalone-wip-zafman-testing2-distro-basic-smithi/2687446/remote/smithi168/coredump/1529622699.99940.core >> ... >> warning: .dynamic section for "/lib64/libudev.so.1" is not at the >> expected address (wrong library or version mismatch?) >> [Thread debugging using libthread_db enabled] >> Using host libthread_db library "/lib64/libthread_db.so.1". >> Core was generated by `ceph-osd -i 3 >> --fsid=f2babf17-f782-48ce-9563-54ab3eb8dc70 --auth-supported=none'. >> Program terminated with signal 6, Aborted. >> #0 0x00007f9d992b659b in raise () from /lib64/libpthread.so.0 >> Missing separate debuginfos, use: debuginfo-install >> bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.170-4.el7.x86_64 >> elfutils-libs-0.170-4.el7.x86_64 fuse-libs-2.9.2-10.el7.x86_64 >> glibc-2.17-222.el7.x86_64 gperftools-libs-2.6.1-1.el7.x86_64 >> keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-19.el7.x86_64 >> leveldb-1.12.0-11.el7.x86_64 libaio-0.3.109-13.el7.x86_64 >> libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-52.el7.x86_64 >> libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-12.el7_5.x86_64 >> libgcc-4.8.5-28.el7_5.1.x86_64 libibverbs-15-7.el7_5.x86_64 >> libnl3-3.2.28-4.el7.x86_64 liboath-2.4.1-9.el7.x86_64 >> libselinux-2.5-12.el7.x86_64 libstdc++-4.8.5-28.el7_5.1.x86_64 >> libuuid-2.23.2-52.el7.x86_64 lttng-ust-2.4.1-4.el7.x86_64 >> lz4-1.7.5-2.el7.x86_64 nspr-4.19.0-1.el7_5.x86_64 >> nss-3.36.0-5.el7_5.x86_64 nss-softokn-3.36.0-5.el7_5.x86_64 >> nss-softokn-freebl-3.36.0-5.el7_5.x86_64 nss-util-3.36.0-1.el7_5.x86_64 >> openssl-libs-1.0.2k-12.el7.x86_64 pcre-8.32-17.el7.x86_64 >> snappy-1.1.0-3.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 >> systemd-libs-219-57.el7.x86_64 userspace-rcu-0.7.16-1.el7.x86_64 >> xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64 >> (gdb) thread >> [Current thread is 1 (Thread 0x7f9d7671f700 (LWP 100040))] >> (gdb) bt >> #0 0x00007f9d992b659b in raise () from /lib64/libpthread.so.0 >> #1 0x0000558705d5f521 in reraise_fatal (signum=6) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/global/signal_handler.cc:74 >> #2 handle_fatal_signal (signum=6) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/global/signal_handler.cc:138 >> #3 <signal handler called> >> #4 0x00007f9d982d6277 in raise () from /lib64/libc.so.6 >> #5 0x00007f9d982d7968 in abort () from /lib64/libc.so.6 >> #6 0x00007f9d98be5ac5 in __cxa_vec_dtor () from /lib64/libstdc++.so.6 >> #7 0x00007f9d98be3a63 in ?? () from /lib64/libstdc++.so.6 >> #8 0x00007f9d76719a20 in ?? () >> #9 0x0000000000000ae0 in ?? () >> #10 0x00007f9d9b1fe5c6 in (anonymous namespace)::do_memalign(unsigned >> long, unsigned long) () from /lib64/libtcmalloc.so.4 >> #11 0x00007f9d9b21d010 in tc_posix_memalign () from /lib64/libtcmalloc.so.4 >> #12 0x00007f9d9b20aacc in tcmalloc::PageHeap::Carve(tcmalloc::Span*, >> unsigned long) () from /lib64/libtcmalloc.so.4 >> #13 0x00007f9d9b20b591 in tcmalloc::PageHeap::New(unsigned long) () from >> /lib64/libtcmalloc.so.4 >> #14 0x00007f9d9b20a230 in tcmalloc::CentralFreeList::Populate() () from >> /lib64/libtcmalloc.so.4 >> #15 0x00007f9d767199d0 in ?? () >> #16 0x0000000000000000 in ?? () >> >> # gdb /usr/bin/ceph-osd >> /ceph/teuthology-archive/dzafman-2018-06-21_15:29:45-rados:standalone-wip-zafman-testing2-distro-basic-smithi/2687443/remote/smithi111/coredump/1529622850.110914.core >> >> Thread 1 (Thread 0x7fe4bf8e2700 (LWP 111012)): >> #0 0x00007fe4e247959b in raise () from /lib64/libpthread.so.0 >> #1 0x0000562bfbd41521 in reraise_fatal (signum=6) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/global/signal_handler.cc:74 >> #2 handle_fatal_signal (signum=6) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/global/signal_handler.cc:138 >> #3 <signal handler called> >> #4 0x00007fe4e1499277 in raise () from /lib64/libc.so.6 >> #5 0x00007fe4e149a968 in abort () from /lib64/libc.so.6 >> #6 0x00007fe4e1da8ac5 in __cxa_vec_dtor () from /lib64/libstdc++.so.6 >> #7 0x00007fe4e1da6a63 in ?? () from /lib64/libstdc++.so.6 >> #8 0x00007fe4bf8dca20 in ?? () >> #9 0x0000000000000ae0 in ?? () >> #10 0x00007fe4e43c15c6 in (anonymous namespace)::do_memalign(unsigned >> long, unsigned long) () from /lib64/libtcmalloc.so.4 >> #11 0x00007fe4e43e0010 in tc_posix_memalign () from /lib64/libtcmalloc.so.4 >> #12 0x00007fe4e58cbc23 in raw (mempool=10, l=2710, c=<optimized out>, >> this=0x8) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/include/buffer_raw.h:44 >> #13 raw_combined (mempool=10, align=2710, l=2710, dataptr=<optimized >> out>, this=0x8) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/buffer.cc:181 >> #14 create (mempool=10, align=2710, len=2710) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/buffer.cc:214 >> #15 ceph::buffer::create_aligned_in_mempool (len=2710, align=2710, >> mempool=10) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/buffer.cc:709 >> #16 0x0cb4eb4a50de6700 in ?? () >> #17 0x0000562bfc425940 in ?? () >> #18 0x0000562bfbb5a030 in FileStore::read (this=<optimized out>, ch=..., >> oid=..., offset=<optimized out>, len=<optimized out>, bl=..., >> op_flags=4222983760) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/os/filestore/FileStore.cc:3382 >> #19 0x0000000000000a96 in ?? () >> #20 0x0000562bfbb59e50 in ?? () at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/os/filestore/FileStore.cc:1909 >> #21 0x0000562bfb8053ed in OSDService::_get_map_bl (this=0x3a, >> e=1356752640, bl=...) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:1364 >> #22 0x0000562bfb812866 in OSDService::try_get_map (this=0x562bfdca6360, >> this@entry=0x562bfdb433e0, epoch=<optimized out>, epoch@entry=58) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:1480 >> #23 0x0000562bfb81b2dd in OSD::advance_pg >> (this=this@entry=0x562bfdb42000, osd_epoch=<optimized out>, >> pg=pg@entry=0x562bfdc34000, handle=..., rctx=rctx@entry=0x7fe4bf8dcf90) >> at /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:7747 >> #24 0x0000562bfb81ba81 in OSD::dequeue_peering_evt (this=0x562bfdb42000, >> sdata=<optimized out>, pg=0x562bfdc34000, evt=std::shared_ptr (count 2, >> weak 0) 0x562bfdc40f90, handle=...) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:8877 >> #25 0x0000562bfba741a0 in PGPeeringItem::run (this=<optimized out>, >> osd=<optimized out>, sdata=<optimized out>, pg=..., handle=...) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OpQueueItem.cc:34 >> #26 0x0000562bfb824532 in run (handle=..., pg=..., sdata=<optimized >> out>, osd=<optimized out>, this=0x7fe4bf8dd140) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OpQueueItem.h:134 >> #27 OSD::ShardedOpWQ::_process (this=0x562bfdb43048, >> thread_index=<optimized out>, hb=<optimized out>) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/osd/OSD.cc:9849 >> #28 0x00007fe4e5923f63 in ShardedThreadPool::shardedthreadpool_worker >> (this=0x562bfdb42930, thread_index=<optimized out>) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/WorkQueue.cc:339 >> #29 0x00007fe4e5924b50 in ShardedThreadPool::WorkThreadSharded::entry >> (this=<optimized out>) at >> /usr/src/debug/ceph-14.0.0-696-gd12ea8b/src/common/WorkQueue.h:690 >> #30 0x00007fe4e2471e25 in start_thread () from /lib64/libpthread.so.0 >> #31 0x00007fe4e1561bad in clone () from /lib64/libc.so.6 >> >> David >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Cheers, Brad -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
