Re: Luminous OSD startup errors

[Abhishek <abhishek@xxxxxxxx>]

On 2017-08-15 15:38, Andras Pataki wrote:
> Thanks for the quick response and the pointer.  The dev build fixed the 
> issue.
>
> Andras
>
>
> On 08/15/2017 09:19 AM, Jason Dillaman wrote:
> > I believe this is a known issue [1] and that there will potentially be
> > a new 12.1.4 RC released because of it. The tracker ticket has a link
> > to a set of development packages that should resolve the issue in the
> > meantime.

We've just started building packages for 12.1.4, so we should be able to 
get this out of the door soon

Best,
Abhishek


> > [1] http://tracker.ceph.com/issues/20985
> >
> > On Tue, Aug 15, 2017 at 9:08 AM, Andras Pataki
> > <apataki@xxxxxxxxxxxxxxxxxxxxx> wrote:
> > > After upgrading to the latest Luminous RC (12.1.3), all our OSD's are
> > > crashing with the following assert:
> > >
> > >       0> 2017-08-15 08:28:49.479238 7f9b7615cd00 -1
> > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.3/rpm/el7/BUILD/ceph-12.1.3/src/osd/PGLog.h:
> > > In function 'static void PGLog::read_log_and_missing(ObjectStore*, 
> > > coll_t,
> > > coll_t, ghobject_t, const pg_info_t&, PGLog::IndexedLog&, 
> > > missing_type&,
> > > bool, std::ostringstream&, bool, bool*, const DoutPrefixProvider*,
> > > std::set<std::basic_string<char> >*, bool) [with missing_type =
> > > pg_missing_set<true>; std::ostringstream = 
> > > std::basic_ostringstream<char>]'
> > > thread 7f9b7615cd00 time 2017-08-15 08:28:49.477367
> > > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.3/rpm/el7/BUILD/ceph-12.1.3/src/osd/PGLog.h:
> > > 1301: FAILED assert(force_rebuild_missing)
> > >
> > >   ceph version 12.1.3 (c56d9c07b342c08419bbc18dcf2a4c5fae62b9cf) 
> > > luminous
> > > (rc)
> > >   1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> > > const*)+0x110) [0x55d0f2be3b50]
> > >   2: (void PGLog::read_log_and_missing<pg_missing_set<true> 
> > > >(ObjectStore*,
> > > coll_t, coll_t, ghobject_t, pg_info_t const&, PGLog::IndexedLog&,
> > > pg_missing_set<true>&, bool, std::basic_ostringstream<char,
> > > std::char_traits<char>, std::allocator<char> >&, bool, bool*,
> > > DoutPrefixProvider const*, std::set<std::string, 
> > > std::less<std::string>,
> > > std::allocator<std::string> >*, bool)+0x773) [0x55d0f276f013]
> > >   3: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x52b)
> > > [0x55d0f272739b]
> > >   4: (OSD::load_pgs()+0x97a) [0x55d0f2673dea]
> > >   5: (OSD::init()+0x2179) [0x55d0f268c319]
> > >   6: (main()+0x2def) [0x55d0f2591ccf]
> > >   7: (__libc_start_main()+0xf5) [0x7f9b727d6b35]
> > >   8: (()+0x4ac006) [0x55d0f2630006]
> > >
> > > Looking at the code in PGLog.h, the change from 12.1.2 to 12.1.3 (in
> > > read_log_missing) was:
> > >
> > >          if (p->key() == "divergent_priors") {
> > >            ::decode(divergent_priors, bp);
> > >            ldpp_dout(dpp, 20) << "read_log_and_missing " <<
> > > divergent_priors.size()
> > >                               << " divergent_priors" << dendl;
> > >            has_divergent_priors = true;
> > >            debug_verify_stored_missing = false;
> > >
> > > to
> > >
> > >          if (p->key() == "divergent_priors") {
> > >            ::decode(divergent_priors, bp);
> > >            ldpp_dout(dpp, 20) << "read_log_and_missing " <<
> > > divergent_priors.size()
> > >                               << " divergent_priors" << dendl;
> > >            assert(force_rebuild_missing);
> > >            debug_verify_stored_missing = false;
> > >
> > > and it seems like force_rebuild_missing is not being set.
> > >
> > > This cluster was upgraded from Jewel to 12.1.1, then 12.1.2 and now 
> > > 12.1.3.
> > > So it seems something didn't happen correctly during the upgrade.  
> > > Any ideas
> > > how to fix it?
> > >
> > > Andras
> > >
> > >
> > > _______________________________________________
> > > ceph-users mailing list
> > > ceph-users@xxxxxxxxxxxxxx
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com