Hello,
Can anyone help me interpret the below error, which one of our OSDs has been occasionally throwing since last night.
Thanks.
-----
Jan 29 03:00:53 osd-host ceph-osd[10964]: 2018-01-29 03:00:53.509185 7fe4ae431700 -1 bluestore(/var/lib/ceph/osd/ceph-9) _balance_bluefs_freespace allocate failed on 0x80000000 min_alloc_size 0x4000
Jan 29 03:00:57 osd-host ceph-osd[10964]: /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_balance_bluefs_freespace(PExtentVector*)' thread 7fe4ae431700 time 2018-01-29 03:00:57.736207
Jan 29 03:00:57 osd-host ceph-osd[10964]: /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 4939: FAILED assert(0 == "allocate failed, wtf")
Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55e88d2ae892]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >*)+0x1b21) [0x55e88d1405c1]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (()+0x76da) [0x7fe4bec766da]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (clone()+0x5f) [0x7fe4bdce8d7f]
Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jan 29 03:00:57 osd-host ceph-osd[10964]: 2018-01-29 03:00:57.741058 7fe4ae431700 -1 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_balance_bluefs_freespace(PExtentVector*)' thread 7fe4ae431700 time 201
8-01-29 03:00:57.736207
Jan 29 03:00:57 osd-host ceph-osd[10964]: /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 4939: FAILED assert(0 == "allocate failed, wtf")
Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55e88d2ae892]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >*)+0x1b21) [0x55e88d1405c1]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (()+0x76da) [0x7fe4bec766da]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (clone()+0x5f) [0x7fe4bdce8d7f]
Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jan 29 03:00:57 osd-host ceph-osd[10964]: 0> 2018-01-29 03:00:57.741058 7fe4ae431700 -1 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::_balance_bluefs_freespace(PExtentVector*)' thread 7fe4ae431700
time 2018-01-29 03:00:57.736207
Jan 29 03:00:57 osd-host ceph-osd[10964]: /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 4939: FAILED assert(0 == "allocate failed, wtf")
Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55e88d2ae892]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >*)+0x1b21) [0x55e88d1405c1]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (()+0x76da) [0x7fe4bec766da]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (clone()+0x5f) [0x7fe4bdce8d7f]
Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jan 29 03:00:57 osd-host ceph-osd[10964]: *** Caught signal (Aborted) **
Jan 29 03:00:57 osd-host ceph-osd[10964]: in thread 7fe4ae431700 thread_name:bstore_kv_sync
Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (()+0xa65824) [0x55e88d26b824]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (()+0x11670) [0x7fe4bec80670]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (gsignal()+0x9f) [0x7fe4bdc1577f]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (abort()+0x16a) [0x7fe4bdc1737a]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55e88d2aea1e]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >*)+0x1b21) [0x55e88d1405c1]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 7: (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 8: (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 9: (()+0x76da) [0x7fe4bec766da]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 10: (clone()+0x5f) [0x7fe4bdce8d7f]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 2018-01-29 03:00:57.790980 7fe4ae431700 -1 *** Caught signal (Aborted) **
Jan 29 03:00:57 osd-host ceph-osd[10964]: in thread 7fe4ae431700 thread_name:bstore_kv_sync
Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (()+0xa65824) [0x55e88d26b824]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (()+0x11670) [0x7fe4bec80670]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (gsignal()+0x9f) [0x7fe4bdc1577f]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (abort()+0x16a) [0x7fe4bdc1737a]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55e88d2aea1e]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >*)+0x1b21) [0x55e88d1405c1]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 7: (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 8: (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 9: (()+0x76da) [0x7fe4bec766da]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 10: (clone()+0x5f) [0x7fe4bdce8d7f]
Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jan 29 03:00:57 osd-host ceph-osd[10964]: 0> 2018-01-29 03:00:57.790980 7fe4ae431700 -1 *** Caught signal (Aborted) **
Jan 29 03:00:57 osd-host ceph-osd[10964]: in thread 7fe4ae431700 thread_name:bstore_kv_sync
Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (()+0xa65824) [0x55e88d26b824]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (()+0x11670) [0x7fe4bec80670]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (gsignal()+0x9f) [0x7fe4bdc1577f]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (abort()+0x16a) [0x7fe4bdc1737a]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x55e88d2aea1e]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >*)+0x1b21) [0x55e88d1405c1]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 7: (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 8: (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 9: (()+0x76da) [0x7fe4bec766da]
Jan 29 03:00:57 osd-host ceph-osd[10964]: 10: (clone()+0x5f) [0x7fe4bdce8d7f]
Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jan 29 03:00:58 osd-host systemd[1]: ceph-osd@9.service: Main process exited, code=killed, status=6/ABRT
-----
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
