StupidAllocator::get_fragmentation crash (was: Re: mimic OSD won't start)

[Gregory Farnum <gfarnum@xxxxxxxxxx>]

Looks like this was a crash in BlueStore. See if you get more useful
information about the problem by turning on the osd with "debug
bluestore = 20" in its config file, and perhaps create a ticket in the
BlueStore component at tracker.ceph.com.
-Greg

On Wed, Aug 15, 2018 at 8:14 AM, Ben Gao <bengao168@xxxxxxx> wrote:
> Hi, I have 4 host with 4 osd running ceph mimic for couple weeks. It became failing with three of them today. Just tried to restart them but no chance for running. Any suggestions on the issues?
>
>
>
> Thanks.
>
>
>
>
>
>
>
> 8月 10 23:47:48 test87 systemd[1]: ceph-osd@0.service: main process exited, code=killed, status=6/ABRT
>
>
>
> 8月 10 23:47:48 test87 systemd[1]: Unit ceph-osd@0.service entered failed state.
>
>
>
> 8月 10 23:47:48 test87 systemd[1]: ceph-osd@0.service failed.
>
>
>
> 8月 10 23:48:08 test87 systemd[1]: ceph-osd@0.service holdoff time over, scheduling restart.
>
>
>
> 8月 10 23:48:08 test87 systemd[1]: Starting Ceph object storage daemon osd.0...
>
>
>
> 8月 10 23:48:09 test87 systemd[1]: Started Ceph object storage daemon osd.0.
>
>
>
> 8月 10 23:48:09 test87 ceph-osd[2406302]: starting osd.0 at - osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7f568617653f]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 2: (()+0x286727) [0x7f5686176727]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 3: (StupidAllocator::get_fragmentation(unsigned long)+0x265) [0x96bac453a5]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 4: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x7a9) [0x96bab34b49]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 5: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x9a) [0x96bab4238a]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 6: (BlueStore::_kv_finalize_thread()+0x41e) [0x96bab433be]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 7: (BlueStore::KVFinalizeThread::entry()+0xd) [0x96bab9585d]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 8: (()+0x7e25) [0x7f5682e5ce25]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 9: (clone()+0x6d) [0x7f5681f5034d]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 2018-08-10 23:48:11.939 7f566f2fc700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7f568617653f]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 2: (()+0x286727) [0x7f5686176727]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 3: (StupidAllocator::get_fragmentation(unsigned long)+0x265) [0x96bac453a5]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 4: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x7a9) [0x96bab34b49]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 5: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x9a) [0x96bab4238a]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 6: (BlueStore::_kv_finalize_thread()+0x41e) [0x96bab433be]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 7: (BlueStore::KVFinalizeThread::entry()+0xd) [0x96bab9585d]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 8: (()+0x7e25) [0x7f5682e5ce25]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 9: (clone()+0x6d) [0x7f5681f5034d]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 0> 2018-08-10 23:48:11.939 7f566f2fc700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/D
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7f568617653f]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 2: (()+0x286727) [0x7f5686176727]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 3: (StupidAllocator::get_fragmentation(unsigned long)+0x265) [0x96bac453a5]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 4: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x7a9) [0x96bab34b49]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 5: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x9a) [0x96bab4238a]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 6: (BlueStore::_kv_finalize_thread()+0x41e) [0x96bab433be]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 7: (BlueStore::KVFinalizeThread::entry()+0xd) [0x96bab9585d]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 8: (()+0x7e25) [0x7f5682e5ce25]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 9: (clone()+0x6d) [0x7f5681f5034d]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: *** Caught signal (Aborted) **
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: in thread 7f566f2fc700 thread_name:bstore_kv_final
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 1: (()+0x8e1870) [0x96bac6f870]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 2: (()+0xf5e0) [0x7f5682e645e0]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 3: (gsignal()+0x37) [0x7f5681e8d1f7]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 4: (abort()+0x148) [0x7f5681e8e8e8]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x25d) [0x7f568617669d]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 6: (()+0x286727) [0x7f5686176727]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 7: (StupidAllocator::get_fragmentation(unsigned long)+0x265) [0x96bac453a5]
>
>
>
> 8月 10 23:48:11 test87 ceph-osd[2406302]: 8: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x7a9) [0x96bab34b49]
>
>
>
>
>
>
>
>