On Tue, Jan 30, 2018 at 7:31 AM, Sergey Malinin <hell@xxxxxxxxxxx> wrote: > Hello, > Can anyone help me interpret the below error, which one of our OSDs has been > occasionally throwing since last night. > Thanks. See http://tracker.ceph.com/issues/22510. > > ----- > Jan 29 03:00:53 osd-host ceph-osd[10964]: 2018-01-29 03:00:53.509185 > 7fe4ae431700 -1 bluestore(/var/lib/ceph/osd/ceph-9) > _balance_bluefs_freespace allocate failed on 0x80000000 min_alloc_size > 0x4000 > Jan 29 03:00:57 osd-host ceph-osd[10964]: > /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In function 'int > BlueStore::_balance_bluefs_freespace(PExtentVector*)' thread 7fe4ae431700 > time 2018-01-29 03:00:57.736207 > Jan 29 03:00:57 osd-host ceph-osd[10964]: > /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 4939: FAILED assert(0 == > "allocate failed, wtf") > Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 > (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) > Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x102) [0x55e88d2ae892] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: > (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, > mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >>*)+0x1b21) [0x55e88d1405c1] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: > (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: > (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (()+0x76da) [0x7fe4bec766da] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (clone()+0x5f) > [0x7fe4bdce8d7f] > Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Jan 29 03:00:57 osd-host ceph-osd[10964]: 2018-01-29 03:00:57.741058 > 7fe4ae431700 -1 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In > function 'int BlueStore::_balance_bluefs_freespace(PExtentVector*)' thread > 7fe4ae431700 time 201 > 8-01-29 03:00:57.736207 > Jan 29 03:00:57 osd-host ceph-osd[10964]: > /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 4939: FAILED assert(0 == > "allocate failed, wtf") > Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 > (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) > Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x102) [0x55e88d2ae892] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: > (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, > mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >>*)+0x1b21) [0x55e88d1405c1] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: > (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: > (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (()+0x76da) [0x7fe4bec766da] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (clone()+0x5f) > [0x7fe4bdce8d7f] > Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Jan 29 03:00:57 osd-host ceph-osd[10964]: 0> 2018-01-29 03:00:57.741058 > 7fe4ae431700 -1 /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: In > function 'int BlueStore::_balance_bluefs_freespace(PExtentVector*)' thread > 7fe4ae431700 > time 2018-01-29 03:00:57.736207 > Jan 29 03:00:57 osd-host ceph-osd[10964]: > /build/ceph-12.2.2/src/os/bluestore/BlueStore.cc: 4939: FAILED assert(0 == > "allocate failed, wtf") > Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 > (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) > Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x102) [0x55e88d2ae892] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: > (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, > mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >>*)+0x1b21) [0x55e88d1405c1] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: > (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: > (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (()+0x76da) [0x7fe4bec766da] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: (clone()+0x5f) > [0x7fe4bdce8d7f] > Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Jan 29 03:00:57 osd-host ceph-osd[10964]: *** Caught signal (Aborted) ** > Jan 29 03:00:57 osd-host ceph-osd[10964]: in thread 7fe4ae431700 > thread_name:bstore_kv_sync > Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 > (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) > Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (()+0xa65824) [0x55e88d26b824] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (()+0x11670) [0x7fe4bec80670] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (gsignal()+0x9f) > [0x7fe4bdc1577f] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (abort()+0x16a) > [0x7fe4bdc1737a] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x28e) [0x55e88d2aea1e] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: > (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, > mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >>*)+0x1b21) [0x55e88d1405c1] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 7: > (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 8: > (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 9: (()+0x76da) [0x7fe4bec766da] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 10: (clone()+0x5f) > [0x7fe4bdce8d7f] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 2018-01-29 03:00:57.790980 > 7fe4ae431700 -1 *** Caught signal (Aborted) ** > Jan 29 03:00:57 osd-host ceph-osd[10964]: in thread 7fe4ae431700 > thread_name:bstore_kv_sync > Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 > (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) > Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (()+0xa65824) [0x55e88d26b824] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (()+0x11670) [0x7fe4bec80670] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (gsignal()+0x9f) > [0x7fe4bdc1577f] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (abort()+0x16a) > [0x7fe4bdc1737a] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x28e) [0x55e88d2aea1e] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: > (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, > mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >>*)+0x1b21) [0x55e88d1405c1] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 7: > (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 8: > (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 9: (()+0x76da) [0x7fe4bec766da] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 10: (clone()+0x5f) > [0x7fe4bdce8d7f] > Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Jan 29 03:00:57 osd-host ceph-osd[10964]: 0> 2018-01-29 03:00:57.790980 > 7fe4ae431700 -1 *** Caught signal (Aborted) ** > Jan 29 03:00:57 osd-host ceph-osd[10964]: in thread 7fe4ae431700 > thread_name:bstore_kv_sync > Jan 29 03:00:57 osd-host ceph-osd[10964]: ceph version 12.2.2 > (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable) > Jan 29 03:00:57 osd-host ceph-osd[10964]: 1: (()+0xa65824) [0x55e88d26b824] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 2: (()+0x11670) [0x7fe4bec80670] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 3: (gsignal()+0x9f) > [0x7fe4bdc1577f] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 4: (abort()+0x16a) > [0x7fe4bdc1737a] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 5: (ceph::__ceph_assert_fail(char > const*, char const*, int, char const*)+0x28e) [0x55e88d2aea1e] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 6: > (BlueStore::_balance_bluefs_freespace(std::vector<bluestore_pextent_t, > mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> >>*)+0x1b21) [0x55e88d1405c1] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 7: > (BlueStore::_kv_sync_thread()+0x1ac0) [0x55e88d143040] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 8: > (BlueStore::KVSyncThread::entry()+0xd) [0x55e88d186f8d] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 9: (()+0x76da) [0x7fe4bec766da] > Jan 29 03:00:57 osd-host ceph-osd[10964]: 10: (clone()+0x5f) > [0x7fe4bdce8d7f] > Jan 29 03:00:57 osd-host ceph-osd[10964]: NOTE: a copy of the executable, > or `objdump -rdS <executable>` is needed to interpret this. > Jan 29 03:00:58 osd-host systemd[1]: ceph-osd@9.service: Main process > exited, code=killed, status=6/ABRT > ----- > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Cheers, Brad _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |