Hi, I have 4 host with 4 osd running ceph mimic for couple weeks. It became failing with three of them today. Just tried to restart them but no chance for running. Any suggestions on the issues? Thanks. 8月 10 23:47:48 test87 systemd[1]: ceph-osd@0.service: main process exited, code=killed, status=6/ABRT 8月 10 23:47:48 test87 systemd[1]: Unit ceph-osd@0.service entered failed state. 8月 10 23:47:48 test87 systemd[1]: ceph-osd@0.service failed. 8月 10 23:48:08 test87 systemd[1]: ceph-osd@0.service holdoff time over, scheduling restart. 8月 10 23:48:08 test87 systemd[1]: Starting Ceph object storage daemon osd.0... 8月 10 23:48:09 test87 systemd[1]: Started Ceph object storage daemon osd.0. 8月 10 23:48:09 test87 ceph-osd[2406302]: starting osd.0 at - osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal 8月 10 23:48:11 test87 ceph-osd[2406302]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2. 8月 10 23:48:11 test87 ceph-osd[2406302]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2. 8月 10 23:48:11 test87 ceph-osd[2406302]: ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 8月 10 23:48:11 test87 ceph-osd[2406302]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7f568617653f] 8月 10 23:48:11 test87 ceph-osd[2406302]: 2: (()+0x286727) [0x7f5686176727] 8月 10 23:48:11 test87 ceph-osd[2406302]: 3: (StupidAllocator::get_fragmentation(unsigned long)+0x265) [0x96bac453a5] 8月 10 23:48:11 test87 ceph-osd[2406302]: 4: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x7a9) [0x96bab34b49] 8月 10 23:48:11 test87 ceph-osd[2406302]: 5: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x9a) [0x96bab4238a] 8月 10 23:48:11 test87 ceph-osd[2406302]: 6: (BlueStore::_kv_finalize_thread()+0x41e) [0x96bab433be] 8月 10 23:48:11 test87 ceph-osd[2406302]: 7: (BlueStore::KVFinalizeThread::entry()+0xd) [0x96bab9585d] 8月 10 23:48:11 test87 ceph-osd[2406302]: 8: (()+0x7e25) [0x7f5682e5ce25] 8月 10 23:48:11 test87 ceph-osd[2406302]: 9: (clone()+0x6d) [0x7f5681f5034d] 8月 10 23:48:11 test87 ceph-osd[2406302]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 8月 10 23:48:11 test87 ceph-osd[2406302]: 2018-08-10 23:48:11.939 7f566f2fc700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST 8月 10 23:48:11 test87 ceph-osd[2406302]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2. 8月 10 23:48:11 test87 ceph-osd[2406302]: ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 8月 10 23:48:11 test87 ceph-osd[2406302]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7f568617653f] 8月 10 23:48:11 test87 ceph-osd[2406302]: 2: (()+0x286727) [0x7f5686176727] 8月 10 23:48:11 test87 ceph-osd[2406302]: 3: (StupidAllocator::get_fragmentation(unsigned long)+0x265) [0x96bac453a5] 8月 10 23:48:11 test87 ceph-osd[2406302]: 4: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x7a9) [0x96bab34b49] 8月 10 23:48:11 test87 ceph-osd[2406302]: 5: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x9a) [0x96bab4238a] 8月 10 23:48:11 test87 ceph-osd[2406302]: 6: (BlueStore::_kv_finalize_thread()+0x41e) [0x96bab433be] 8月 10 23:48:11 test87 ceph-osd[2406302]: 7: (BlueStore::KVFinalizeThread::entry()+0xd) [0x96bab9585d] 8月 10 23:48:11 test87 ceph-osd[2406302]: 8: (()+0x7e25) [0x7f5682e5ce25] 8月 10 23:48:11 test87 ceph-osd[2406302]: 9: (clone()+0x6d) [0x7f5681f5034d] 8月 10 23:48:11 test87 ceph-osd[2406302]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 8月 10 23:48:11 test87 ceph-osd[2406302]: 0> 2018-08-10 23:48:11.939 7f566f2fc700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/D 8月 10 23:48:11 test87 ceph-osd[2406302]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2. 8月 10 23:48:11 test87 ceph-osd[2406302]: ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 8月 10 23:48:11 test87 ceph-osd[2406302]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xff) [0x7f568617653f] 8月 10 23:48:11 test87 ceph-osd[2406302]: 2: (()+0x286727) [0x7f5686176727] 8月 10 23:48:11 test87 ceph-osd[2406302]: 3: (StupidAllocator::get_fragmentation(unsigned long)+0x265) [0x96bac453a5] 8月 10 23:48:11 test87 ceph-osd[2406302]: 4: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x7a9) [0x96bab34b49] 8月 10 23:48:11 test87 ceph-osd[2406302]: 5: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x9a) [0x96bab4238a] 8月 10 23:48:11 test87 ceph-osd[2406302]: 6: (BlueStore::_kv_finalize_thread()+0x41e) [0x96bab433be] 8月 10 23:48:11 test87 ceph-osd[2406302]: 7: (BlueStore::KVFinalizeThread::entry()+0xd) [0x96bab9585d] 8月 10 23:48:11 test87 ceph-osd[2406302]: 8: (()+0x7e25) [0x7f5682e5ce25] 8月 10 23:48:11 test87 ceph-osd[2406302]: 9: (clone()+0x6d) [0x7f5681f5034d] 8月 10 23:48:11 test87 ceph-osd[2406302]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 8月 10 23:48:11 test87 ceph-osd[2406302]: *** Caught signal (Aborted) ** 8月 10 23:48:11 test87 ceph-osd[2406302]: in thread 7f566f2fc700 thread_name:bstore_kv_final 8月 10 23:48:11 test87 ceph-osd[2406302]: ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable) 8月 10 23:48:11 test87 ceph-osd[2406302]: 1: (()+0x8e1870) [0x96bac6f870] 8月 10 23:48:11 test87 ceph-osd[2406302]: 2: (()+0xf5e0) [0x7f5682e645e0] 8月 10 23:48:11 test87 ceph-osd[2406302]: 3: (gsignal()+0x37) [0x7f5681e8d1f7] 8月 10 23:48:11 test87 ceph-osd[2406302]: 4: (abort()+0x148) [0x7f5681e8e8e8] 8月 10 23:48:11 test87 ceph-osd[2406302]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x25d) [0x7f568617669d] 8月 10 23:48:11 test87 ceph-osd[2406302]: 6: (()+0x286727) [0x7f5686176727] 8月 10 23:48:11 test87 ceph-osd[2406302]: 7: (StupidAllocator::get_fragmentation(unsigned long)+0x265) [0x96bac453a5] 8月 10 23:48:11 test87 ceph-osd[2406302]: 8: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x7a9) [0x96bab34b49]
