Hello, I'm having this same issue on 12.2.8. Should I repoen the bug report? This cluster started on 12.2.4 and was upgraded to 12.2.5 and then directly to 12.2.8 (we skipped 2.6 and 2.7) but the malfunctioning OSD is on a new node installed with 12.2.8. We're using CentOS 7.5, and bluestore for ceph. This particular node has SSD disks. I have an extract of the log and objdump if needed. Thanks, El Miércoles 11/07/2018 a las 18:31, Gregory Farnum escribió: > A bit delayed, but Radoslaw looked at this some and has a diagnosis on the > tracker ticket: http://tracker.ceph.com/issues/24715 > So it looks like a symptom of a bug that was already fixed for unrelated > reasons. :) > -Greg > > On Wed, Jun 27, 2018 at 4:51 AM Dyweni - Ceph-Users > <6EXbab4FYk8H@xxxxxxxxxx> > > wrote: > > Good Morning, > > > > I have rebuilt the OSD and the cluster is healthy now. > > > > I have one pool with 3 replica setup. I am a bit concerned that > > removing a snapshot can cause an OSD to crash. I've asked myself what > > would have happened if 2 OSD's had crashed? God forbid, what if 3 or > > more OSD's had crashed with this same error? How would I have recovered > > from that? > > > > So planning for the future: > > > > 1. Is there any way to proactively scan for (and even repair) this? > > > > 2. What could have caused this? > > > > We experienced a cluster wide power outage lasting several hours several > > days ago. The outage occurred at a time when no snapshots were being > > created. The cluster was brought back up in a controlled manner and no > > errors were discovered immediately afterward (Ceph reported healthly). > > Could this have caused corruption? > > > > Thanks, > > Dyweni > > > > On 2018-06-25 09:34, Dyweni - Ceph-Users wrote: > > > Hi, > > > > > > Is there any information you'd like to grab off this OSD? Anything I > > > can provide to help you troubleshoot this? > > > > > > I ask, because if not, I'm going to reformat / rebuild this OSD > > > (unless there is a faster way to repair this issue). > > > > > > Thanks, > > > Dyweni > > > > > > On 2018-06-25 07:30, Dyweni - Ceph-Users wrote: > > >> Good Morning, > > >> > > >> After removing roughly 20-some rbd shapshots, one of my OSD's has > > >> begun flapping. > > >> > > >> > > >> **** ERROR 1 **** > > >> > > >> 2018-06-25 06:46:39.132257 a0ce2700 -1 osd.8 pg_epoch: 44738 pg[4.e8( > > >> v 44721'485588 (44697'484015,44721'485588] local-lis/les=44593/44595 > > >> n=2972 ec=9422/9422 lis/c 44593/44593 les/c/f 44595/44595/40729 > > >> 44593/44593/44593) [8,7,10] r=0 lpr=44593 crt=44721'485588 lcod > > >> 44721'485586 mlcod 44721'485586 active+clean+snapt > > >> rim snaptrimq=[276~1,280~1,2af~1,2e8~4]] removing snap head > > >> 2018-06-25 06:46:41.314172 a1ce2700 -1 > > > > /var/tmp/portage/sys-cluster/ceph-12.2.5/work/ceph-12.2.5/src/os/bluestore/bluestore_types.cc: > > >> In function 'void bluestore_extent_ref_map_t::put(uint64_t, uint32_t, > > >> PExtentVector*, bool*)' thread a1ce2700 time 2018-06-25 > > >> 06:46:41.220388 > > > > /var/tmp/portage/sys-cluster/ceph-12.2.5/work/ceph-12.2.5/src/os/bluestore/bluestore_types.cc: > > >> 217: FAILED assert(0 == "put on missing extent (nothing before)") > > >> > > >> ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) > > >> luminous (stable) > > >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > > >> const*)+0x1bc) [0x2a2c314] > > >> 2: (bluestore_extent_ref_map_t::put(unsigned long long, unsigned int, > > >> std::vector<bluestore_pextent_t, > > >> mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> > > >> > > >> >*, bool*)+0x128) [0x2893650] > > >> > > >> 3: (BlueStore::SharedBlob::put_ref(unsigned long long, unsigned int, > > >> std::vector<bluestore_pextent_t, > > >> mempool::pool_allocator<(mempool::pool_index_t)4, bluestore_pextent_t> > > >> > > >> >*, std::set<BlueStore::SharedBlob*, > > >> > std::less<BlueStore::SharedBlob*>, > > > > std::allocator<BlueStore::SharedBlob*> >*)+0xb8) [0x2791bdc] > > > > >> 4: (BlueStore::_wctx_finish(BlueStore::TransContext*, > > >> boost::intrusive_ptr<BlueStore::Collection>&, > > >> boost::intrusive_ptr<BlueStore::Onode>, BlueStore::WriteContext*, > > >> std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, > > >> std::allocator<BlueStore::SharedBlob*> >*)+0x5c8) [0x27f3254] > > >> 5: (BlueStore::_do_truncate(BlueStore::TransContext*, > > >> boost::intrusive_ptr<BlueStore::Collection>&, > > >> boost::intrusive_ptr<BlueStore::Onode>, unsigned long long, > > >> std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, > > >> std::allocator<BlueStore::SharedBlob*> >*)+0x360) [0x27f7834] > > >> 6: (BlueStore::_do_remove(BlueStore::TransContext*, > > >> boost::intrusive_ptr<BlueStore::Collection>&, > > >> boost::intrusive_ptr<BlueStore::Onode>)+0xb4) [0x27f81b4] > > >> 7: (BlueStore::_remove(BlueStore::TransContext*, > > >> boost::intrusive_ptr<BlueStore::Collection>&, > > >> boost::intrusive_ptr<BlueStore::Onode>&)+0x1dc) [0x27f9638] > > >> 8: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, > > >> ObjectStore::Transaction*)+0xe7c) [0x27e855c] > > >> 9: (BlueStore::queue_transactions(ObjectStore::Sequencer*, > > >> std::vector<ObjectStore::Transaction, > > >> std::allocator<ObjectStore::Transaction> >&, > > >> boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x67c) > > >> [0x27e6f80] > > >> 10: (ObjectStore::queue_transactions(ObjectStore::Sequencer*, > > >> std::vector<ObjectStore::Transaction, > > >> std::allocator<ObjectStore::Transaction> >&, Context*, Context*, > > >> Context*, boost::intrusive_ptr<TrackedOp>, > > >> ThreadPool::TPHandle*)+0x118) [0x1f9ce48] > > >> 11: > > >> (PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction > > >>, std::allocator<ObjectStore::Transaction> >&, > > >> boost::intrusive_ptr<OpRequest>)+0x9c) [0x22dd754] > > >> 12: (ReplicatedBackend::submit_transaction(hobject_t const&, > > >> object_stat_sum_t const&, eversion_t const&, > > >> std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, > > >> eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, > > >> std::allocator<pg_log_entry_t> > const&, > > >> boost::optional<pg_hit_set_histo > > >> ry_t>&, Context*, Context*, Context*, unsigned long long, osd_reqid_t, > > >> boost::intrusive_ptr<OpRequest>)+0x6f4) [0x25c0568] > > >> 13: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, > > >> PrimaryLogPG::OpContext*)+0x7f4) [0x228ac98] > > >> 14: > > > > (PrimaryLogPG::simple_opc_submit(std::unique_ptr<PrimaryLogPG::OpContext, > > > > >> std::default_delete<PrimaryLogPG::OpContext> >)+0x1b8) [0x228bc54] > > >> 15: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork > > >> const&)+0x1970) [0x22c5d4c] > > >> 16: (boost::statechart::detail::reaction_result > > > > boost::statechart::custom_reaction<PrimaryLogPG::DoSnapWork>::react<Prima > >ryLogPG::AwaitAsyncWork, > > > > >> boost::statechart::event_base, void > > >> const*>(PrimaryLogPG::AwaitAsyncWork&, boost::statechart::event_base > > >> const&, void const* const&)+0x58) [0x23b245c] > > >> 17: (boost::statechart::detail::reaction_result > > >> boost::statechart::simple_state<PrimaryLogPG::AwaitAsyncWork, > > >> PrimaryLogPG::Trimming, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > > >> > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_: > > >> :na, mpl_::na, mpl_::na, mpl_::na>, > > > > (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_re > >act_impl<boost::mpl::list<boost::statechart::custom_reaction<PrimaryLogPG: > >:DoSnapWork>, > > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, > > >> boost::statechart::simple_state<PrimaryLogPG::AwaitAsyncWork, > > >> PrimaryLogPG::Trimming, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl > > >> _::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, > > >> (boost::statechart::history_mode)0> > > >> > > >> >(boost::statechart::simple_state<PrimaryLogPG::AwaitAsyncWork, > > > > PrimaryLogPG::Trimming, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, > > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > > mpl_::na, mpl_::na > > > > >> , mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > > >> mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>&, > > >> boost::statechart::event_base const&, void const*)+0x30) [0x23b0f04] > > >> 18: (boost::statechart::detail::reaction_result > > >> boost::statechart::simple_state<PrimaryLogPG::AwaitAsyncWork, > > >> PrimaryLogPG::Trimming, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > > >> > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_: > > >> :na, mpl_::na, mpl_::na, mpl_::na>, > > > > (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost:: > >statechart::custom_reaction<PrimaryLogPG::DoSnapWork>, > > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > > >> mpl_::na, mpl > > >> _::na, mpl_::na, mpl_::na, mpl_::na> >(boost::statechart::event_base > > >> const&, void const*)+0x28) [0x23af7cc] > > >> 19: (boost::statechart::simple_state<PrimaryLogPG::AwaitAsyncWork, > > >> PrimaryLogPG::Trimming, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, > > >> > > >> mpl_::na, mpl_::na, mpl_::na>, (boost: > > >> :statechart::history_mode)0>::react_impl(boost::statechart::event_base > > >> > > >> const&, void const*)+0x28) [0x23ad744] > > >> 20: > > > > (boost::statechart::detail::send_function<boost::statechart::detail::stat > >e_base<std::allocator<void>, > > > > >> boost::statechart::detail::rtti_policy>, > > >> boost::statechart::event_base, void const*>::operator()()+0x40) > > >> [0x21b6000] > > >> 21: (boost::statechart::detail::reaction_result > > > > boost::statechart::null_exception_translator::operator()<boost::statechar > >t::detail::send_function<boost::statechart::detail::state_base<std::alloca > >tor<void>, > > > > >> boost::statechart::detail::rtti_policy>, > > >> boost::statechart::event_base, void const*>, boost::statechart::state > > >> _machine<PrimaryLogPG::SnapTrimmer, PrimaryLogPG::NotTrimming, > > >> std::allocator<void>, > > > > boost::statechart::null_exception_translator>::exception_event_handler>(b > >oost::statechart::detail::send_function<boost::statechart::detail::state_b > >ase<std::allocator<void>, > > > > >> boost::statechart::detail::rtti_policy>, boost::statechart: > > >> :event_base, void const*>, > > >> > > >> boost::statechart::state_machine<PrimaryLogPG::SnapTrimmer, > > >> PrimaryLogPG::NotTrimming, std::allocator<void>, > > > > boost::statechart::null_exception_translator>::exception_event_handler)+0 > >x24) > > > > >> [0x233c4a4] > > >> 22: (boost::statechart::state_machine<PrimaryLogPG::SnapTrimmer, > > >> PrimaryLogPG::NotTrimming, std::allocator<void>, > > > > boost::statechart::null_exception_translator>::send_event(boost::statecha > >rt::event_base > > > > >> const&)+0x13c) [0x2316328] > > >> 23: (boost::statechart::state_machine<PrimaryLogPG::SnapTrimmer, > > >> PrimaryLogPG::NotTrimming, std::allocator<void>, > > > > boost::statechart::null_exception_translator>::process_event(boost::state > >chart::event_base > > > > >> const&)+0x38) [0x22f6404] > > >> 24: (PrimaryLogPG::snap_trimmer(unsigned int)+0x1a8) [0x2258bb0] > > >> 25: (PGQueueable::RunVis::operator()(PGSnapTrim const&)+0x40) > > >> [0x2520cb4] > > >> 26: (boost::disable_if_c<(false)&&boost::is_same<PGSnapTrim&, > > >> PGSnapTrim&>::value, void>::type > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>::internal_visit<PGSnapTrim&>(PGSnapTrim&, int)+0x2c) > > >> [0x209f6bc] > > >> 27: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>::result_type > > > > boost::detail::variant::visitation_impl_invoke_impl<boost::detail::varian > >t::invoke_visitor<PGQueueable::RunVis, > > > > >> false>, void*, PGSnapTrim>(int, > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis, false>&, > > >> void*, PGSnapTrim* > > >> , mpl_::bool_<true>)+0x38) [0x2095a74] > > >> 28: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>::result_type > > > > boost::detail::variant::visitation_impl_invoke<boost::detail::variant::in > >voke_visitor<PGQueueable::RunVis, > > > > >> false>, void*, PGSnapTrim, > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, PGScrub, > > >> PGRecovery>::has_fallback_ty > > >> pe_>(int, boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>&, void*, PGSnapTrim*, > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, PGScrub, > > >> PGRecovery>::has_fallback_type_, int)+0x4c) [0x2086f30] > > >> 29: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>::result_type > > >> boost::detail::variant::visitation_impl<mpl_::int_<0>, > > > > boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mp > >l::l_item<mpl_::long_<4l>, > > > > >> boost::intrusive_ptr<OpRequest>, boost::mpl::l_item<mpl_::long_<3l>, > > >> PGSnapT > > >> rim, boost::mpl::l_item<mpl_::long_<2l>, PGScrub, > > >> boost::mpl::l_item<mpl_::long_<1l>, PGRecovery, boost::mpl::l_end> > > > > >> > > >> > >, boost::mpl::l_iter<boost::mpl::l_end> >, > > > > boost::detail::variant::invoke_visitor<PGQueueable::RunVis, false>, > > void*, boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, > > PGScrub, PGReco > > > > >> very>::has_fallback_type_>(int, int, > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis, false>&, > > >> void*, mpl_::bool_<false>, > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, PGScrub, > > >> PGRecovery>::has_fallback_type_, mpl_::int_<0>*, > > >> boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<bo > > >> ost::mpl::l_item<mpl_::long_<4l>, boost::intrusive_ptr<OpRequest>, > > >> boost::mpl::l_item<mpl_::long_<3l>, PGSnapTrim, > > >> boost::mpl::l_item<mpl_::long_<2l>, PGScrub, > > >> boost::mpl::l_item<mpl_::long_<1l>, PGRecovery, boost::mpl::l_end> > > > > >> > > >> > >, boost::mpl::l_iter<boost::mpl::l_end> >*)+0xf0) [0x20710fc] > > >> > > >> 30: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>::result_type boost::variant<boost::intrusive_ptr<OpRequest>, > > >> PGSnapTrim, PGScrub, > > > > PGRecovery>::internal_apply_visitor_impl<boost::detail::variant::invoke_v > >isitor<PGQueueable::RunVis, > > > > >> false>, void*>(int, int, boost::detail::variant::invoke_visit > > >> or<PGQueueable::RunVis, false>&, void*)+0x60) [0x20513e8] > > >> 31: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>::result_type boost::variant<boost::intrusive_ptr<OpRequest>, > > >> PGSnapTrim, PGScrub, > > > > PGRecovery>::internal_apply_visitor<boost::detail::variant::invoke_visito > >r<PGQueueable::RunVis, > > > > >> false> >(boost::detail::variant::invoke_visitor<PGQueueable::RunVi > > >> s, false>&)+0x4c) [0x20254c4] > > >> 32: (PGQueueable::RunVis::result_type > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, PGScrub, > > >> PGRecovery>::apply_visitor<PGQueueable::RunVis>(PGQueueable::RunVis&) > > >> &+0x4c) [0x1ff7798] > > >> 33: (PGQueueable::RunVis::result_type > > >> boost::apply_visitor<PGQueueable::RunVis, > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, PGScrub, > > >> PGRecovery>&>(PGQueueable::RunVis&, > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, PGScrub, > > >> PGRecovery>&)+0x2c) [0x1fcbb70] > > >> 34: (PGQueueable::run(OSD*, boost::intrusive_ptr<PG>&, > > >> ThreadPool::TPHandle&)+0x5c) [0x1fa76c8] > > >> 35: (OSD::ShardedOpWQ::_process(unsigned int, > > >> ceph::heartbeat_handle_d*)+0x23e4) [0x1f7f7bc] > > >> 36: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x508) > > >> [0x2a30f2c] > > >> 37: (ShardedThreadPool::WorkThreadSharded::entry()+0x2c) [0x2a32a68] > > >> 38: (Thread::entry_wrapper()+0xf4) [0x2c39e34] > > >> 39: (Thread::_entry_func(void*)+0x18) [0x2c39d28] > > >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is > > >> needed to interpret this. > > >> > > >> > > >> > > >> > > >> **** ERROR 2 (repeats each time the OSD tries to restart) **** > > >> > > >> 2018-06-25 06:47:54.618917 a1cee700 -1 > > >> bluestore(/var/lib/ceph/osd/ceph-8).collection(4.7d_head 0xad81ce0) > > >> load_shared_blob sbid 0x52044 not found at key 0x0000000000052044 > > >> 2018-06-25 06:47:54.689880 a1cee700 -1 > > > > /var/tmp/portage/sys-cluster/ceph-12.2.5/work/ceph-12.2.5/src/os/bluestore/BlueStore.cc: > > >> In function 'void > > >> BlueStore::Collection::load_shared_blob(BlueStore::SharedBlobRef)' > > >> thread a1cee700 time 2018-06-25 06:47:54.619013 > > > > /var/tmp/portage/sys-cluster/ceph-12.2.5/work/ceph-12.2.5/src/os/bluestore/BlueStore.cc: > > >> 3158: FAILED assert(0 == "uh oh, missing shared_blob") > > >> > > >> ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) > > >> luminous (stable) > > >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > > >> const*)+0x1bc) [0x2a20314] > > >> 2: > > > > (BlueStore::Collection::load_shared_blob(boost::intrusive_ptr<BlueStore:: > >SharedBlob>)+0x32c) > > > > >> [0x27937a8] > > >> 3: (BlueStore::_wctx_finish(BlueStore::TransContext*, > > >> boost::intrusive_ptr<BlueStore::Collection>&, > > >> boost::intrusive_ptr<BlueStore::Onode>, BlueStore::WriteContext*, > > >> std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, > > >> std::allocator<BlueStore::SharedBlob*> >*)+0x4e0) [0x27e716c] > > >> 4: (BlueStore::_do_truncate(BlueStore::TransContext*, > > >> boost::intrusive_ptr<BlueStore::Collection>&, > > >> boost::intrusive_ptr<BlueStore::Onode>, unsigned long long, > > >> std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, > > >> std::allocator<BlueStore::SharedBlob*> >*)+0x360) [0x27eb834] > > >> 5: (BlueStore::_do_remove(BlueStore::TransContext*, > > >> boost::intrusive_ptr<BlueStore::Collection>&, > > >> boost::intrusive_ptr<BlueStore::Onode>)+0xb4) [0x27ec1b4] > > >> 6: (BlueStore::_remove(BlueStore::TransContext*, > > >> boost::intrusive_ptr<BlueStore::Collection>&, > > >> boost::intrusive_ptr<BlueStore::Onode>&)+0x1dc) [0x27ed638] > > >> 7: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, > > >> ObjectStore::Transaction*)+0xe7c) [0x27dc55c] > > >> 8: (BlueStore::queue_transactions(ObjectStore::Sequencer*, > > >> std::vector<ObjectStore::Transaction, > > >> std::allocator<ObjectStore::Transaction> >&, > > >> boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x67c) > > >> [0x27daf80] > > >> 9: (ObjectStore::queue_transactions(ObjectStore::Sequencer*, > > >> std::vector<ObjectStore::Transaction, > > >> std::allocator<ObjectStore::Transaction> >&, Context*, Context*, > > >> Context*, boost::intrusive_ptr<TrackedOp>, > > >> ThreadPool::TPHandle*)+0x118) [0x1f90e48] > > >> 10: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, > > >> ObjectStore::Transaction&&, Context*, Context*, Context*, > > >> boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0xc0) > > >> [0x1f90cbc] > > >> 11: (PrimaryLogPG::remove_missing_object(hobject_t const&, > > >> eversion_t, Context*)+0x3b0) [0x228adbc] > > >> 12: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, > > >> PGBackend::RecoveryHandle*)+0x348) [0x22899b4] > > >> 13: (PrimaryLogPG::recover_primary(unsigned long long, > > >> ThreadPool::TPHandle&)+0x1488) [0x2296ba0] > > >> 14: (PrimaryLogPG::start_recovery_ops(unsigned long long, > > >> ThreadPool::TPHandle&, unsigned long long*)+0x2b8) [0x2294054] > > >> 15: (OSD::do_recovery(PG*, unsigned int, unsigned long long, > > >> ThreadPool::TPHandle&)+0x574) [0x1f6abe4] > > >> 16: (PGQueueable::RunVis::operator()(PGRecovery const&)+0x60) > > >> [0x2514d68] > > >> 17: (boost::disable_if_c<(false)&&boost::is_same<PGRecovery&, > > >> PGRecovery&>::value, void>::type > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>::internal_visit<PGRecovery&>(PGRecovery&, int)+0x2c) > > >> [0x2093724] > > >> 18: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>::result_type > > > > boost::detail::variant::visitation_impl_invoke_impl<boost::detail::varian > >t::invoke_visitor<PGQueueable::RunVis, > > > > >> false>, void*, PGRecovery>(int, > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis, false>&, > > >> void*, PGRecovery*, mpl_::bool_<true>)+0x38) [0x2089af4] > > >> 19: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>::result_type > > > > boost::detail::variant::visitation_impl_invoke<boost::detail::variant::in > >voke_visitor<PGQueueable::RunVis, > > > > >> false>, void*, PGRecovery, > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, PGScrub, > > >> PGRecovery>::has_fallback_type_>(int, > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis, false>&, > > >> void*, PGRecovery*, boost::variant<boost::intrusive_ptr<OpRequest>, > > >> PGSnapTrim, PGScrub, PGRecovery>::has_fallback_type_, int)+0x4c) > > >> [0x207b020] > > >> 20: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>::result_type > > >> boost::detail::variant::visitation_impl<mpl_::int_<0>, > > > > boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mp > >l::l_item<mpl_::long_<4l>, > > > > >> boost::intrusive_ptr<OpRequest>, boost::mpl::l_item<mpl_::long_<3l>, > > >> PGSnapTrim, boost::mpl::l_item<mpl_::long_<2l>, PGScrub, > > >> boost::mpl::l_item<mpl_::long_<1l>, PGRecovery, boost::mpl::l_end> > > > > >> > > >> > >, boost::mpl::l_iter<boost::mpl::l_end> >, > > > > boost::detail::variant::invoke_visitor<PGQueueable::RunVis, false>, > > void*, boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, > > PGScrub, PGRecovery>::has_fallback_type_>(int, int, > > boost::detail::variant::invoke_visitor<PGQueueable::RunVis, false>&, > > void*, mpl_::bool_<false>, > > boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, PGScrub, > > PGRecovery>::has_fallback_type_, mpl_::int_<0>*, > > boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mp > >l::l_item<mpl_::long_<4l>, boost::intrusive_ptr<OpRequest>, > > boost::mpl::l_item<mpl_::long_<3l>, PGSnapTrim, > > boost::mpl::l_item<mpl_::long_<2l>, PGScrub, > > boost::mpl::l_item<mpl_::long_<1l>, PGRecovery, boost::mpl::l_end> > > > > > >, boost::mpl::l_iter<boost::mpl::l_end> >*)+0x140) [0x206514c] > > > > >> 21: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>::result_type boost::variant<boost::intrusive_ptr<OpRequest>, > > >> PGSnapTrim, PGScrub, > > > > PGRecovery>::internal_apply_visitor_impl<boost::detail::variant::invoke_v > >isitor<PGQueueable::RunVis, > > > > >> false>, void*>(int, int, > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis, false>&, > > >> void*)+0x60) [0x20453e8] > > >> 22: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>::result_type boost::variant<boost::intrusive_ptr<OpRequest>, > > >> PGSnapTrim, PGScrub, > > > > PGRecovery>::internal_apply_visitor<boost::detail::variant::invoke_visito > >r<PGQueueable::RunVis, > > > > >> false> >(boost::detail::variant::invoke_visitor<PGQueueable::RunVis, > > >> false>&)+0x4c) [0x20194c4] > > >> 23: (PGQueueable::RunVis::result_type > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, PGScrub, > > >> PGRecovery>::apply_visitor<PGQueueable::RunVis>(PGQueueable::RunVis&) > > >> &+0x4c) [0x1feb798] > > >> 24: (PGQueueable::RunVis::result_type > > >> boost::apply_visitor<PGQueueable::RunVis, > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, PGScrub, > > >> PGRecovery>&>(PGQueueable::RunVis&, > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, PGScrub, > > >> PGRecovery>&)+0x2c) [0x1fbfb70] > > >> 25: (PGQueueable::run(OSD*, boost::intrusive_ptr<PG>&, > > >> ThreadPool::TPHandle&)+0x5c) [0x1f9b6c8] > > >> 26: (OSD::ShardedOpWQ::_process(unsigned int, > > >> ceph::heartbeat_handle_d*)+0x23e4) [0x1f737bc] > > >> 27: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x508) > > >> [0x2a24f2c] > > >> 28: (ShardedThreadPool::WorkThreadSharded::entry()+0x2c) [0x2a26a68] > > >> 29: (Thread::entry_wrapper()+0xf4) [0x2c2de34] > > >> 30: (Thread::_entry_func(void*)+0x18) [0x2c2dd28] > > >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is > > >> needed to interpret this. > > >> > > >> > > >> Thanks, > > >> Dyweni -- Ricardo J. Barberis Usuario Linux Nº 250625: http://counter.li.org/ Usuario LFS Nº 5121: http://www.linuxfromscratch.org/ Senior SysAdmin / IT Architect - www.DonWeb.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |