Re: Ceph 12.2.5 - FAILED assert(0 == "put on missing extent (nothing before)")

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I created https://tracker.ceph.com/issues/36303

I can wait maybe a couple of days before recreating this OSD if you need me to 
extract som more info.

Thanks.

El Miércoles 03/10/2018 a las 01:43, Gregory Farnum escribió:
> I'd create a new ticket and reference the older one; they may not have the
> same cause.
>
> On Tue, Oct 2, 2018 at 12:33 PM Ricardo J. Barberis <ricardo@xxxxxxxxxxxxx>
>
> wrote:
> > Hello,
> >
> > I'm having this same issue on 12.2.8. Should I repoen the bug report?
> >
> > This cluster started on 12.2.4 and was upgraded to 12.2.5 and then
> > directly to
> > 12.2.8 (we skipped 2.6 and 2.7) but the malfunctioning OSD is on a new
> > node
> > installed with 12.2.8.
> >
> > We're using CentOS 7.5, and bluestore for ceph. This particular node has
> > SSD
> > disks.
> >
> > I have an extract of the log and objdump if needed.
> >
> > Thanks,
> >
> > El Miércoles 11/07/2018 a las 18:31, Gregory Farnum escribió:
> > > A bit delayed, but Radoslaw looked at this some and has a diagnosis on
> >
> > the
> >
> > > tracker ticket: http://tracker.ceph.com/issues/24715
> > > So it looks like a symptom of a bug that was already fixed for
> > > unrelated reasons. :)
> > > -Greg
> > >
> > > On Wed, Jun 27, 2018 at 4:51 AM Dyweni - Ceph-Users
> > > <6EXbab4FYk8H@xxxxxxxxxx>
> > >
> > > wrote:
> > > > Good Morning,
> > > >
> > > > I have rebuilt the OSD and the cluster is healthy now.
> > > >
> > > > I have one pool with 3 replica setup.  I am a bit concerned that
> > > > removing a snapshot can cause an OSD to crash.  I've asked myself
> > > > what would have happened if 2 OSD's had crashed?  God forbid, what if
> > > > 3 or more OSD's had crashed with this same error?  How would I have
> >
> > recovered
> >
> > > > from that?
> > > >
> > > > So planning for the future:
> > > >
> > > >   1. Is there any way to proactively scan for (and even repair) this?
> > > >
> > > >   2. What could have caused this?
> > > >
> > > > We experienced a cluster wide power outage lasting several hours
> >
> > several
> >
> > > > days ago.  The outage occurred at a time when no snapshots were being
> > > > created.  The cluster was brought back up in a controlled manner and
> > > > no errors were discovered immediately afterward (Ceph reported
> > > > healthly). Could this have caused corruption?
> > > >
> > > > Thanks,
> > > > Dyweni
> > > >
> > > > On 2018-06-25 09:34, Dyweni - Ceph-Users wrote:
> > > > > Hi,
> > > > >
> > > > > Is there any information you'd like to grab off this OSD?  Anything
> > > > > I can provide to help you troubleshoot this?
> > > > >
> > > > > I ask, because if not, I'm going to reformat / rebuild this OSD
> > > > > (unless there is a faster way to repair this issue).
> > > > >
> > > > > Thanks,
> > > > > Dyweni
> > > > >
> > > > > On 2018-06-25 07:30, Dyweni - Ceph-Users wrote:
> > > > >> Good Morning,
> > > > >>
> > > > >> After removing roughly 20-some rbd shapshots, one of my OSD's has
> > > > >> begun flapping.
> > > > >>
> > > > >>
> > > > >> **** ERROR 1 ****
> > > > >>
> > > > >> 2018-06-25 06:46:39.132257 a0ce2700 -1 osd.8 pg_epoch: 44738
> >
> > pg[4.e8(
> >
> > > > >> v 44721'485588 (44697'484015,44721'485588]
> > > > >> local-lis/les=44593/44595 n=2972 ec=9422/9422 lis/c 44593/44593
> > > > >> les/c/f 44595/44595/40729 44593/44593/44593) [8,7,10] r=0
> > > > >> lpr=44593 crt=44721'485588 lcod 44721'485586 mlcod 44721'485586
> > > > >> active+clean+snapt
> > > > >> rim snaptrimq=[276~1,280~1,2af~1,2e8~4]] removing snap head
> > > > >> 2018-06-25 06:46:41.314172 a1ce2700 -1
> >
> 
> /var/tmp/portage/sys-cluster/ceph-12.2.5/work/ceph-12.2.5/src/os/bluestore/bluestore_types.cc:
> > > > >> In function 'void bluestore_extent_ref_map_t::put(uint64_t,
> >
> > uint32_t,
> >
> > > > >> PExtentVector*, bool*)' thread a1ce2700 time 2018-06-25
> > > > >> 06:46:41.220388
> >
> 
> /var/tmp/portage/sys-cluster/ceph-12.2.5/work/ceph-12.2.5/src/os/bluestore/bluestore_types.cc:
> > > > >> 217: FAILED assert(0 == "put on missing extent (nothing before)")
> > > > >>
> > > > >>  ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a)
> > > > >> luminous (stable)
> > > > >>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> > > > >> const*)+0x1bc) [0x2a2c314]
> > > > >>  2: (bluestore_extent_ref_map_t::put(unsigned long long, unsigned
> >
> > int,
> >
> > > > >> std::vector<bluestore_pextent_t,
> > > > >> mempool::pool_allocator<(mempool::pool_index_t)4,
> >
> > bluestore_pextent_t>
> >
> > > > >> >*, bool*)+0x128) [0x2893650]
> > > > >>
> > > > >>  3: (BlueStore::SharedBlob::put_ref(unsigned long long, unsigned
> >
> > int,
> >
> > > > >> std::vector<bluestore_pextent_t,
> > > > >> mempool::pool_allocator<(mempool::pool_index_t)4,
> >
> > bluestore_pextent_t>
> >
> > > > >> >*, std::set<BlueStore::SharedBlob*,
> > > > >> > std::less<BlueStore::SharedBlob*>,
> > > >
> > > > std::allocator<BlueStore::SharedBlob*> >*)+0xb8) [0x2791bdc]
> > > >
> > > > >>  4: (BlueStore::_wctx_finish(BlueStore::TransContext*,
> > > > >> boost::intrusive_ptr<BlueStore::Collection>&,
> > > > >> boost::intrusive_ptr<BlueStore::Onode>, BlueStore::WriteContext*,
> > > > >> std::set<BlueStore::SharedBlob*,
> > > > >> std::less<BlueStore::SharedBlob*>,
> > > > >> std::allocator<BlueStore::SharedBlob*> >*)+0x5c8) [0x27f3254] 5:
> > > > >> (BlueStore::_do_truncate(BlueStore::TransContext*,
> > > > >> boost::intrusive_ptr<BlueStore::Collection>&,
> > > > >> boost::intrusive_ptr<BlueStore::Onode>, unsigned long long,
> > > > >> std::set<BlueStore::SharedBlob*,
> > > > >> std::less<BlueStore::SharedBlob*>,
> > > > >> std::allocator<BlueStore::SharedBlob*> >*)+0x360) [0x27f7834] 6:
> > > > >> (BlueStore::_do_remove(BlueStore::TransContext*,
> > > > >> boost::intrusive_ptr<BlueStore::Collection>&,
> > > > >> boost::intrusive_ptr<BlueStore::Onode>)+0xb4) [0x27f81b4]
> > > > >>  7: (BlueStore::_remove(BlueStore::TransContext*,
> > > > >> boost::intrusive_ptr<BlueStore::Collection>&,
> > > > >> boost::intrusive_ptr<BlueStore::Onode>&)+0x1dc) [0x27f9638]
> > > > >>  8: (BlueStore::_txc_add_transaction(BlueStore::TransContext*,
> > > > >> ObjectStore::Transaction*)+0xe7c) [0x27e855c]
> > > > >>  9: (BlueStore::queue_transactions(ObjectStore::Sequencer*,
> > > > >> std::vector<ObjectStore::Transaction,
> > > > >> std::allocator<ObjectStore::Transaction> >&,
> > > > >> boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x67c)
> > > > >> [0x27e6f80]
> > > > >>  10: (ObjectStore::queue_transactions(ObjectStore::Sequencer*,
> > > > >> std::vector<ObjectStore::Transaction,
> > > > >> std::allocator<ObjectStore::Transaction> >&, Context*, Context*,
> > > > >> Context*, boost::intrusive_ptr<TrackedOp>,
> > > > >> ThreadPool::TPHandle*)+0x118) [0x1f9ce48]
> > > > >>  11:
> >
> > (PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction
> >
> > > > >>, std::allocator<ObjectStore::Transaction> >&,
> > > > >> boost::intrusive_ptr<OpRequest>)+0x9c) [0x22dd754]
> > > > >>  12: (ReplicatedBackend::submit_transaction(hobject_t const&,
> > > > >> object_stat_sum_t const&, eversion_t const&,
> > > > >> std::unique_ptr<PGTransaction, std::default_delete<PGTransaction>
> > >
> > >&&,
> > >
> > > > >> eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t,
> > > > >> std::allocator<pg_log_entry_t> > const&,
> > > > >> boost::optional<pg_hit_set_histo
> > > > >> ry_t>&, Context*, Context*, Context*, unsigned long long,
> >
> > osd_reqid_t,
> >
> > > > >> boost::intrusive_ptr<OpRequest>)+0x6f4) [0x25c0568]
> > > > >>  13: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*,
> > > > >> PrimaryLogPG::OpContext*)+0x7f4) [0x228ac98]
> > > > >>  14:
> >
> > (PrimaryLogPG::simple_opc_submit(std::unique_ptr<PrimaryLogPG::OpContext,
> >
> > > > >> std::default_delete<PrimaryLogPG::OpContext> >)+0x1b8) [0x228bc54]
> > > > >>  15: (PrimaryLogPG::AwaitAsyncWork::react(PrimaryLogPG::DoSnapWork
> > > > >> const&)+0x1970) [0x22c5d4c]
> > > > >>  16: (boost::statechart::detail::reaction_result
> >
> > boost::statechart::custom_reaction<PrimaryLogPG::DoSnapWork>::react<Prima
> >
> > > >ryLogPG::AwaitAsyncWork,
> > > >
> > > > >> boost::statechart::event_base, void
> > > > >> const*>(PrimaryLogPG::AwaitAsyncWork&,
> > > > >> boost::statechart::event_base const&, void const* const&)+0x58)
> > > > >> [0x23b245c]
> > > > >>  17: (boost::statechart::detail::reaction_result
> > > > >> boost::statechart::simple_state<PrimaryLogPG::AwaitAsyncWork,
> > > > >> PrimaryLogPG::Trimming, boost::mpl::list<mpl_::na, mpl_::na,
> >
> > mpl_::na,
> >
> > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> >
> > mpl_::na,
> >
> > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_:
> > > > >> :na, mpl_::na, mpl_::na, mpl_::na>,
> >
> > (boost::statechart::history_mode)0>::local_react_impl_non_empty::local_re
> >
> > 
>act_impl<boost::mpl::list<boost::statechart::custom_reaction<PrimaryLogPG:
> > > >:DoSnapWork>,
> > > >:
> > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> >
> > mpl_::na,
> >
> > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> >
> > mpl_::na>,
> >
> > > > >> boost::statechart::simple_state<PrimaryLogPG::AwaitAsyncWork,
> > > > >> PrimaryLogPG::Trimming, boost::mpl::list<mpl_::na, mpl_::na,
> >
> > mpl_::na,
> >
> > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> >
> > mpl_::na,
> >
> > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl
> > > > >> _::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>,
> > > > >> (boost::statechart::history_mode)0>
> > > > >>
> > > > >> >(boost::statechart::simple_state<PrimaryLogPG::AwaitAsyncWork,
> > > >
> > > > PrimaryLogPG::Trimming, boost::mpl::list<mpl_::na, mpl_::na,
> > > > mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> > > > mpl_::na, mpl_::na, mpl_::na
> > > >
> > > > >> , mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> > > > >> mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>&,
> > > > >> boost::statechart::event_base const&, void const*)+0x30)
> > > > >> [0x23b0f04] 18: (boost::statechart::detail::reaction_result
> > > > >> boost::statechart::simple_state<PrimaryLogPG::AwaitAsyncWork,
> > > > >> PrimaryLogPG::Trimming, boost::mpl::list<mpl_::na, mpl_::na,
> >
> > mpl_::na,
> >
> > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> >
> > mpl_::na,
> >
> > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_:
> > > > >> :na, mpl_::na, mpl_::na, mpl_::na>,
> >
> > (boost::statechart::history_mode)0>::local_react<boost::mpl::list<boost::
> > > >statechart::custom_reaction<PrimaryLogPG::DoSnapWork>,
> > > >
> > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> >
> > mpl_::na,
> >
> > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> >
> > mpl_::na,
> >
> > > > >> mpl_::na, mpl
> > > > >> _::na, mpl_::na, mpl_::na, mpl_::na>
> > > > >> >(boost::statechart::event_base const&, void const*)+0x28)
> > > > >> [0x23af7cc]
> > > > >>  19:
> > > > >> (boost::statechart::simple_state<PrimaryLogPG::AwaitAsyncWork,
> > > > >> PrimaryLogPG::Trimming, boost::mpl::list<mpl_::na, mpl_::na,
> >
> > mpl_::na,
> >
> > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> >
> > mpl_::na,
> >
> > > > >> mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
> >
> > mpl_::na,
> >
> > > > >> mpl_::na, mpl_::na, mpl_::na>, (boost:
> > :
> > :statechart::history_mode)0>::react_impl(boost::statechart::event_base
> > :
> > > > >> const&, void const*)+0x28) [0x23ad744]
> > > > >>  20:
> >
> > (boost::statechart::detail::send_function<boost::statechart::detail::stat
> >
> > > >e_base<std::allocator<void>,
> > > >
> > > > >> boost::statechart::detail::rtti_policy>,
> > > > >> boost::statechart::event_base, void const*>::operator()()+0x40)
> > > > >> [0x21b6000]
> > > > >>  21: (boost::statechart::detail::reaction_result
> >
> > boost::statechart::null_exception_translator::operator()<boost::statechar
> >
> > >t::detail::send_function<boost::statechart::detail::state_base<std::allo
> > >ca
> > >
> > > >tor<void>,
> > > >
> > > > >> boost::statechart::detail::rtti_policy>,
> > > > >> boost::statechart::event_base, void const*>,
> >
> > boost::statechart::state
> >
> > > > >> _machine<PrimaryLogPG::SnapTrimmer, PrimaryLogPG::NotTrimming,
> > > > >> std::allocator<void>,
> >
> > boost::statechart::null_exception_translator>::exception_event_handler>(b
> >
> > >oost::statechart::detail::send_function<boost::statechart::detail::state
> > >_b
> > >
> > > >ase<std::allocator<void>,
> > > >
> > > > >> boost::statechart::detail::rtti_policy>, boost::statechart:
> > > > >> :event_base, void const*>,
> > > > >>
> > > > >> boost::statechart::state_machine<PrimaryLogPG::SnapTrimmer,
> > > > >> PrimaryLogPG::NotTrimming, std::allocator<void>,
> >
> > boost::statechart::null_exception_translator>::exception_event_handler)+0
> >
> > > >x24)
> > > >
> > > > >> [0x233c4a4]
> > > > >>  22: (boost::statechart::state_machine<PrimaryLogPG::SnapTrimmer,
> > > > >> PrimaryLogPG::NotTrimming, std::allocator<void>,
> >
> > boost::statechart::null_exception_translator>::send_event(boost::statecha
> >
> > > >rt::event_base
> > > >
> > > > >> const&)+0x13c) [0x2316328]
> > > > >>  23: (boost::statechart::state_machine<PrimaryLogPG::SnapTrimmer,
> > > > >> PrimaryLogPG::NotTrimming, std::allocator<void>,
> >
> > boost::statechart::null_exception_translator>::process_event(boost::state
> >
> > > >chart::event_base
> > > >
> > > > >> const&)+0x38) [0x22f6404]
> > > > >>  24: (PrimaryLogPG::snap_trimmer(unsigned int)+0x1a8) [0x2258bb0]
> > > > >>  25: (PGQueueable::RunVis::operator()(PGSnapTrim const&)+0x40)
> > > > >> [0x2520cb4]
> > > > >>  26: (boost::disable_if_c<(false)&&boost::is_same<PGSnapTrim&,
> > > > >> PGSnapTrim&>::value, void>::type
> > > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>::internal_visit<PGSnapTrim&>(PGSnapTrim&, int)+0x2c)
> > > > >> [0x209f6bc]
> > > > >>  27: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>::result_type
> >
> > boost::detail::variant::visitation_impl_invoke_impl<boost::detail::varian
> >
> > > >t::invoke_visitor<PGQueueable::RunVis,
> > > >
> > > > >> false>, void*, PGSnapTrim>(int,
> > > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>&, void*, PGSnapTrim*
> > > > >> , mpl_::bool_<true>)+0x38) [0x2095a74]
> > > > >>  28: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>::result_type
> >
> > boost::detail::variant::visitation_impl_invoke<boost::detail::variant::in
> >
> > > >voke_visitor<PGQueueable::RunVis,
> > > >
> > > > >> false>, void*, PGSnapTrim,
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub, PGRecovery>::has_fallback_ty
> > > > >> pe_>(int,
> >
> > boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> >
> > > > >> false>&, void*, PGSnapTrim*,
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub, PGRecovery>::has_fallback_type_, int)+0x4c) [0x2086f30]
> > > > >>  29: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>::result_type
> > > > >> boost::detail::variant::visitation_impl<mpl_::int_<0>,
> >
> > boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mp
> >
> > > >l::l_item<mpl_::long_<4l>,
> > > >
> > > > >> boost::intrusive_ptr<OpRequest>,
> > > > >> boost::mpl::l_item<mpl_::long_<3l>, PGSnapT
> > > > >> rim, boost::mpl::l_item<mpl_::long_<2l>, PGScrub,
> > > > >> boost::mpl::l_item<mpl_::long_<1l>, PGRecovery, boost::mpl::l_end>
> > > > >>
> > > > >> > >, boost::mpl::l_iter<boost::mpl::l_end> >,
> > > >
> > > > boost::detail::variant::invoke_visitor<PGQueueable::RunVis, false>,
> > > > void*, boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > PGScrub, PGReco
> > > >
> > > > >> very>::has_fallback_type_>(int, int,
> > > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>&, void*, mpl_::bool_<false>,
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub, PGRecovery>::has_fallback_type_, mpl_::int_<0>*,
> > > > >> boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<bo
> > > > >> ost::mpl::l_item<mpl_::long_<4l>, boost::intrusive_ptr<OpRequest>,
> > > > >> boost::mpl::l_item<mpl_::long_<3l>, PGSnapTrim,
> > > > >> boost::mpl::l_item<mpl_::long_<2l>, PGScrub,
> > > > >> boost::mpl::l_item<mpl_::long_<1l>, PGRecovery, boost::mpl::l_end>
> > > > >>
> > > > >> > >, boost::mpl::l_iter<boost::mpl::l_end> >*)+0xf0) [0x20710fc]
> > > > >>
> > > > >>  30: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>::result_type
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub,
> >
> > PGRecovery>::internal_apply_visitor_impl<boost::detail::variant::invoke_v
> >
> > > >isitor<PGQueueable::RunVis,
> > > >
> > > > >> false>, void*>(int, int, boost::detail::variant::invoke_visit
> > > > >> or<PGQueueable::RunVis, false>&, void*)+0x60) [0x20513e8]
> > > > >>  31: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>::result_type
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub,
> >
> > PGRecovery>::internal_apply_visitor<boost::detail::variant::invoke_visito
> >
> > > >r<PGQueueable::RunVis,
> > > >
> > > > >> false> >(boost::detail::variant::invoke_visitor<PGQueueable::RunVi
> > > > >> s, false>&)+0x4c) [0x20254c4]
> > > > >>  32: (PGQueueable::RunVis::result_type
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub,
> >
> > PGRecovery>::apply_visitor<PGQueueable::RunVis>(PGQueueable::RunVis&)
> >
> > > > >> &+0x4c) [0x1ff7798]
> > > > >>  33: (PGQueueable::RunVis::result_type
> > > > >> boost::apply_visitor<PGQueueable::RunVis,
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub, PGRecovery>&>(PGQueueable::RunVis&,
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub, PGRecovery>&)+0x2c) [0x1fcbb70]
> > > > >>  34: (PGQueueable::run(OSD*, boost::intrusive_ptr<PG>&,
> > > > >> ThreadPool::TPHandle&)+0x5c) [0x1fa76c8]
> > > > >>  35: (OSD::ShardedOpWQ::_process(unsigned int,
> > > > >> ceph::heartbeat_handle_d*)+0x23e4) [0x1f7f7bc]
> > > > >>  36: (ShardedThreadPool::shardedthreadpool_worker(unsigned
> >
> > int)+0x508)
> >
> > > > >> [0x2a30f2c]
> > > > >>  37: (ShardedThreadPool::WorkThreadSharded::entry()+0x2c)
> >
> > [0x2a32a68]
> >
> > > > >>  38: (Thread::entry_wrapper()+0xf4) [0x2c39e34]
> > > > >>  39: (Thread::_entry_func(void*)+0x18) [0x2c39d28]
> > > > >>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> > > > >> needed to interpret this.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> **** ERROR 2 (repeats each time the OSD tries to restart) ****
> > > > >>
> > > > >> 2018-06-25 06:47:54.618917 a1cee700 -1
> > > > >> bluestore(/var/lib/ceph/osd/ceph-8).collection(4.7d_head
> > > > >> 0xad81ce0) load_shared_blob sbid 0x52044 not found at key
> > > > >> 0x0000000000052044 2018-06-25 06:47:54.689880 a1cee700 -1
> >
> 
> /var/tmp/portage/sys-cluster/ceph-12.2.5/work/ceph-12.2.5/src/os/bluestore/BlueStore.cc:
> > > > >> In function 'void
> > > > >> BlueStore::Collection::load_shared_blob(BlueStore::SharedBlobRef)'
> > > > >> thread a1cee700 time 2018-06-25 06:47:54.619013
> >
> 
> /var/tmp/portage/sys-cluster/ceph-12.2.5/work/ceph-12.2.5/src/os/bluestore/BlueStore.cc:
> > > > >> 3158: FAILED assert(0 == "uh oh, missing shared_blob")
> > > > >>
> > > > >>  ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a)
> > > > >> luminous (stable)
> > > > >>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> > > > >> const*)+0x1bc) [0x2a20314]
> > > > >>  2:
> >
> > (BlueStore::Collection::load_shared_blob(boost::intrusive_ptr<BlueStore::
> > > >SharedBlob>)+0x32c)
> > > >
> > > > >> [0x27937a8]
> > > > >>  3: (BlueStore::_wctx_finish(BlueStore::TransContext*,
> > > > >> boost::intrusive_ptr<BlueStore::Collection>&,
> > > > >> boost::intrusive_ptr<BlueStore::Onode>, BlueStore::WriteContext*,
> > > > >> std::set<BlueStore::SharedBlob*,
> > > > >> std::less<BlueStore::SharedBlob*>,
> > > > >> std::allocator<BlueStore::SharedBlob*> >*)+0x4e0) [0x27e716c] 4:
> > > > >> (BlueStore::_do_truncate(BlueStore::TransContext*,
> > > > >> boost::intrusive_ptr<BlueStore::Collection>&,
> > > > >> boost::intrusive_ptr<BlueStore::Onode>, unsigned long long,
> > > > >> std::set<BlueStore::SharedBlob*,
> > > > >> std::less<BlueStore::SharedBlob*>,
> > > > >> std::allocator<BlueStore::SharedBlob*> >*)+0x360) [0x27eb834] 5:
> > > > >> (BlueStore::_do_remove(BlueStore::TransContext*,
> > > > >> boost::intrusive_ptr<BlueStore::Collection>&,
> > > > >> boost::intrusive_ptr<BlueStore::Onode>)+0xb4) [0x27ec1b4]
> > > > >>  6: (BlueStore::_remove(BlueStore::TransContext*,
> > > > >> boost::intrusive_ptr<BlueStore::Collection>&,
> > > > >> boost::intrusive_ptr<BlueStore::Onode>&)+0x1dc) [0x27ed638]
> > > > >>  7: (BlueStore::_txc_add_transaction(BlueStore::TransContext*,
> > > > >> ObjectStore::Transaction*)+0xe7c) [0x27dc55c]
> > > > >>  8: (BlueStore::queue_transactions(ObjectStore::Sequencer*,
> > > > >> std::vector<ObjectStore::Transaction,
> > > > >> std::allocator<ObjectStore::Transaction> >&,
> > > > >> boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x67c)
> > > > >> [0x27daf80]
> > > > >>  9: (ObjectStore::queue_transactions(ObjectStore::Sequencer*,
> > > > >> std::vector<ObjectStore::Transaction,
> > > > >> std::allocator<ObjectStore::Transaction> >&, Context*, Context*,
> > > > >> Context*, boost::intrusive_ptr<TrackedOp>,
> > > > >> ThreadPool::TPHandle*)+0x118) [0x1f90e48]
> > > > >>  10: (ObjectStore::queue_transaction(ObjectStore::Sequencer*,
> > > > >> ObjectStore::Transaction&&, Context*, Context*, Context*,
> > > > >> boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0xc0)
> > > > >> [0x1f90cbc]
> > > > >>  11: (PrimaryLogPG::remove_missing_object(hobject_t const&,
> > > > >> eversion_t, Context*)+0x3b0) [0x228adbc]
> > > > >>  12: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t,
> >
> > int,
> >
> > > > >> PGBackend::RecoveryHandle*)+0x348) [0x22899b4]
> > > > >>  13: (PrimaryLogPG::recover_primary(unsigned long long,
> > > > >> ThreadPool::TPHandle&)+0x1488) [0x2296ba0]
> > > > >>  14: (PrimaryLogPG::start_recovery_ops(unsigned long long,
> > > > >> ThreadPool::TPHandle&, unsigned long long*)+0x2b8) [0x2294054]
> > > > >>  15: (OSD::do_recovery(PG*, unsigned int, unsigned long long,
> > > > >> ThreadPool::TPHandle&)+0x574) [0x1f6abe4]
> > > > >>  16: (PGQueueable::RunVis::operator()(PGRecovery const&)+0x60)
> > > > >> [0x2514d68]
> > > > >>  17: (boost::disable_if_c<(false)&&boost::is_same<PGRecovery&,
> > > > >> PGRecovery&>::value, void>::type
> > > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>::internal_visit<PGRecovery&>(PGRecovery&, int)+0x2c)
> > > > >> [0x2093724]
> > > > >>  18: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>::result_type
> >
> > boost::detail::variant::visitation_impl_invoke_impl<boost::detail::varian
> >
> > > >t::invoke_visitor<PGQueueable::RunVis,
> > > >
> > > > >> false>, void*, PGRecovery>(int,
> > > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>&, void*, PGRecovery*, mpl_::bool_<true>)+0x38) [0x2089af4]
> > > > >>  19: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>::result_type
> >
> > boost::detail::variant::visitation_impl_invoke<boost::detail::variant::in
> >
> > > >voke_visitor<PGQueueable::RunVis,
> > > >
> > > > >> false>, void*, PGRecovery,
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub, PGRecovery>::has_fallback_type_>(int,
> > > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>&, void*, PGRecovery*,
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub, PGRecovery>::has_fallback_type_, int)+0x4c) [0x207b020]
> > > > >>  20: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>::result_type
> > > > >> boost::detail::variant::visitation_impl<mpl_::int_<0>,
> >
> > boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mp
> >
> > > >l::l_item<mpl_::long_<4l>,
> > > >
> > > > >> boost::intrusive_ptr<OpRequest>,
> > > > >> boost::mpl::l_item<mpl_::long_<3l>, PGSnapTrim,
> > > > >> boost::mpl::l_item<mpl_::long_<2l>, PGScrub,
> > > > >> boost::mpl::l_item<mpl_::long_<1l>, PGRecovery, boost::mpl::l_end>
> > > > >>
> > > > >> > >, boost::mpl::l_iter<boost::mpl::l_end> >,
> > > >
> > > > boost::detail::variant::invoke_visitor<PGQueueable::RunVis, false>,
> > > > void*, boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > PGScrub, PGRecovery>::has_fallback_type_>(int, int,
> > > > boost::detail::variant::invoke_visitor<PGQueueable::RunVis, false>&,
> > > > void*, mpl_::bool_<false>,
> > > > boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim, PGScrub,
> > > > PGRecovery>::has_fallback_type_, mpl_::int_<0>*,
> >
> > boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mp
> >
> > > >l::l_item<mpl_::long_<4l>, boost::intrusive_ptr<OpRequest>,
> > > > boost::mpl::l_item<mpl_::long_<3l>, PGSnapTrim,
> > > > boost::mpl::l_item<mpl_::long_<2l>, PGScrub,
> > > > boost::mpl::l_item<mpl_::long_<1l>, PGRecovery, boost::mpl::l_end> >
> > > > >
> > > >
> > > > >, boost::mpl::l_iter<boost::mpl::l_end> >*)+0x140) [0x206514c]
> > > > >
> > > > >>  21: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>::result_type
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub,
> >
> > PGRecovery>::internal_apply_visitor_impl<boost::detail::variant::invoke_v
> >
> > > >isitor<PGQueueable::RunVis,
> > > >
> > > > >> false>, void*>(int, int,
> > > > >> boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>&, void*)+0x60) [0x20453e8]
> > > > >>  22: (boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>::result_type
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub,
> >
> > PGRecovery>::internal_apply_visitor<boost::detail::variant::invoke_visito
> >
> > > >r<PGQueueable::RunVis,
> > > >
> > > > >> false>
> > > > >> >(boost::detail::variant::invoke_visitor<PGQueueable::RunVis,
> > > > >> false>&)+0x4c) [0x20194c4]
> > > > >>  23: (PGQueueable::RunVis::result_type
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub,
> >
> > PGRecovery>::apply_visitor<PGQueueable::RunVis>(PGQueueable::RunVis&)
> >
> > > > >> &+0x4c) [0x1feb798]
> > > > >>  24: (PGQueueable::RunVis::result_type
> > > > >> boost::apply_visitor<PGQueueable::RunVis,
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub, PGRecovery>&>(PGQueueable::RunVis&,
> > > > >> boost::variant<boost::intrusive_ptr<OpRequest>, PGSnapTrim,
> > > > >> PGScrub, PGRecovery>&)+0x2c) [0x1fbfb70]
> > > > >>  25: (PGQueueable::run(OSD*, boost::intrusive_ptr<PG>&,
> > > > >> ThreadPool::TPHandle&)+0x5c) [0x1f9b6c8]
> > > > >>  26: (OSD::ShardedOpWQ::_process(unsigned int,
> > > > >> ceph::heartbeat_handle_d*)+0x23e4) [0x1f737bc]
> > > > >>  27: (ShardedThreadPool::shardedthreadpool_worker(unsigned
> >
> > int)+0x508)
> >
> > > > >> [0x2a24f2c]
> > > > >>  28: (ShardedThreadPool::WorkThreadSharded::entry()+0x2c)
> >
> > [0x2a26a68]
> >
> > > > >>  29: (Thread::entry_wrapper()+0xf4) [0x2c2de34]
> > > > >>  30: (Thread::_entry_func(void*)+0x18) [0x2c2dd28]
> > > > >>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> > > > >> needed to interpret this.
> > > > >>
> > > > >>
> > > > >> Thanks,
> > > > >> Dyweni
> >
> > --
> > Ricardo J. Barberis
> > Usuario Linux Nº 250625: http://counter.li.org/
> > Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
> > Senior SysAdmin / IT Architect - www.DonWeb.com
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@xxxxxxxxxxxxxx
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
Ricardo J. Barberis
Usuario Linux Nº 250625: http://counter.li.org/
Usuario LFS Nº 5121: http://www.linuxfromscratch.org/
Senior SysAdmin / IT Architect - www.DonWeb.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux