Re: StupidAllocator::get_fragmentation crash (was: Re: mimic OSD won't start)

[Igor Fedotov <ifedotov@xxxxxxx>]

I fixed the issue in master:

https://github.com/ceph/ceph/pull/23606


Also working with Ben on bringing his cluster back to life ..


Thanks,

Igor


On 8/16/2018 5:55 PM, Gregory Farnum wrote:
> Looks like this was a crash in BlueStore. See if you get more useful
> information about the problem by turning on the osd with "debug
> bluestore = 20" in its config file, and perhaps create a ticket in the
> BlueStore component at tracker.ceph.com.
> -Greg
>
> On Wed, Aug 15, 2018 at 8:14 AM, Ben Gao <bengao168@xxxxxxx> wrote:
> > Hi, I have 4 host with 4 osd running ceph mimic for couple weeks. It became failing with three of them today. Just tried to restart them but no chance for running. Any suggestions on the issues?
> >
> >
> >
> > Thanks.
> >
> >
> >
> >
> >
> >
> >
> > 8月 10 23:47:48 test87 systemd[1]: ceph-osd@0.service: main process exited, code=killed, status=6/ABRT
> >
> >
> >
> > 8月 10 23:47:48 test87 systemd[1]: Unit ceph-osd@0.service entered failed state.
> >
> >
> >
> > 8月 10 23:47:48 test87 systemd[1]: ceph-osd@0.service failed.
> >
> >
> >
> > 8月 10 23:48:08 test87 systemd[1]: ceph-osd@0.service holdoff time over, scheduling restart.
> >
> >
> >
> > 8月 10 23:48:08 test87 systemd[1]: Starting Ceph object storage daemon osd.0...
> >
> >
> >
> > 8月 10 23:48:09 test87 systemd[1]: Started Ceph object storage daemon osd.0.
> >
> >
> >
> > 8月 10 23:48:09 test87 ceph-osd[2406302]: starting osd.0 at - osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7f568617653f]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 2: (()+0x286727) [0x7f5686176727]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 3: (StupidAllocator::get_fragmentation(unsigned long)+0x265) [0x96bac453a5]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 4: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x7a9) [0x96bab34b49]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 5: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x9a) [0x96bab4238a]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 6: (BlueStore::_kv_finalize_thread()+0x41e) [0x96bab433be]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 7: (BlueStore::KVFinalizeThread::entry()+0xd) [0x96bab9585d]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 8: (()+0x7e25) [0x7f5682e5ce25]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 9: (clone()+0x6d) [0x7f5681f5034d]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 2018-08-10 23:48:11.939 7f566f2fc700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7f568617653f]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 2: (()+0x286727) [0x7f5686176727]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 3: (StupidAllocator::get_fragmentation(unsigned long)+0x265) [0x96bac453a5]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 4: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x7a9) [0x96bab34b49]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 5: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x9a) [0x96bab4238a]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 6: (BlueStore::_kv_finalize_thread()+0x41e) [0x96bab433be]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 7: (BlueStore::KVFinalizeThread::entry()+0xd) [0x96bab9585d]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 8: (()+0x7e25) [0x7f5682e5ce25]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 9: (clone()+0x6d) [0x7f5681f5034d]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 0> 2018-08-10 23:48:11.939 7f566f2fc700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/D
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7f568617653f]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 2: (()+0x286727) [0x7f5686176727]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 3: (StupidAllocator::get_fragmentation(unsigned long)+0x265) [0x96bac453a5]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 4: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x7a9) [0x96bab34b49]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 5: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x9a) [0x96bab4238a]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 6: (BlueStore::_kv_finalize_thread()+0x41e) [0x96bab433be]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 7: (BlueStore::KVFinalizeThread::entry()+0xd) [0x96bab9585d]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 8: (()+0x7e25) [0x7f5682e5ce25]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 9: (clone()+0x6d) [0x7f5681f5034d]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: *** Caught signal (Aborted) **
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: in thread 7f566f2fc700 thread_name:bstore_kv_final
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 1: (()+0x8e1870) [0x96bac6f870]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 2: (()+0xf5e0) [0x7f5682e645e0]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 3: (gsignal()+0x37) [0x7f5681e8d1f7]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 4: (abort()+0x148) [0x7f5681e8e8e8]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x25d) [0x7f568617669d]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 6: (()+0x286727) [0x7f5686176727]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 7: (StupidAllocator::get_fragmentation(unsigned long)+0x265) [0x96bac453a5]
> >
> >
> >
> > 8月 10 23:48:11 test87 ceph-osd[2406302]: 8: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x7a9) [0x96bab34b49]