Hi Sam, I've updated the ticket with logs from the wip run. Nick > -----Original Message----- > From: Samuel Just [mailto:sjust@xxxxxxxxxx] > Sent: 15 November 2016 18:30 > To: Nick Fisk <nick@xxxxxxxxxx> > Cc: Ceph Users <ceph-users@xxxxxxxxxxxxxx> > Subject: Re: After OSD Flap - FAILED assert(oi.version == i->first) > > http://tracker.ceph.com/issues/17916 > > I just pushed a branch wip-17916-jewel based on v10.2.3 with some additional debugging. Once it builds, would you be able to start > the afflicted osds with that version of ceph-osd and > > debug osd = 20 > debug ms = 1 > debug filestore = 20 > > and get me the log? > -Sam > > On Tue, Nov 15, 2016 at 2:06 AM, Nick Fisk <nick@xxxxxxxxxx> wrote: > > Hi, > > > > I have two OSD's which are failing with an assert which looks related > > to missing objects. This happened after a large RBD snapshot was > > deleted causing several OSD's to start flapping as they experienced > > high load. Cluster is fully recovered and I don't need any help from a recovery perspective. I'm happy to Zap and recreate OSD's, > which I will probably do in a couple of days time. Or if anybody looks at the error and see's an easy way to get the OSD to start up, then > bonus!!! > > > > However, I thought I would post in case there is any interest in > > trying to diagnose why this happened. There was no power or networking issues and no hard reboot's, so this is purely contained > within the Ceph OSD process. > > > > The objects that it claims are missing are from the RBD that had the > > snapshot deleted. I'm guessing that the last command before the OSD > > died at some point was to delete those two objects which did actually happen, but for some reason the OSD had died before it got > confirmation??? And now it's trying to delete them, but they don't exist. > > > > I have the full debug 20 log, but pretty much all the lines above the > > below snippet just have it deleting thousands of objects without any problems. > > > > Nick > > > > -4> 2016-11-15 09:46:52.061643 7f728f9368c0 20 read_log 6 divergent_priors > > -3> 2016-11-15 09:46:52.061779 7f728f9368c0 10 read_log checking for missing items over interval (0'0,1607344'260104] > > -2> 2016-11-15 09:46:52.069987 7f728f9368c0 15 read_log missing > > 1553246'255377,1:96e51ad6:::rbd_data.6fd18238e1f29.00000000002555c5:head > > -1> 2016-11-15 09:46:52.070007 7f728f9368c0 15 read_log missing > > 1553190'255366,1:96e51ad6:::rbd_data.6fd18238e1f29.00000000002555c5:6c > > 0> 2016-11-15 09:46:52.071471 7f728f9368c0 -1 osd/PGLog.cc: In > > function 'static void PGLog::read_log(ObjectStore*, coll_t, coll_t, > > ghobject_t, const pg_info_t&, std::map<eversion_t, hobject_t>&, > > PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&, const > > DoutPrefixProvider*, std::set<std::__cxx11::basic_string<char> >*)' > > thread 7f728f9368c0 time 2016-11-15 09:46:52.070023 > > osd/PGLog.cc: 1047: FAILED assert(oi.version == i->first) > > > > ceph version 10.2.3 (ecc23778eb545d8dd55e2e4735b53cc93f92e65b) > > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > > const*)+0x80) [0x5642d2734ea0] > > 2: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, > > pg_info_t const&, std::map<eversion_t, hobject_t, > > std::less<eversion_t>, std::allocator<std::pair<eversion_t const, > > hobject_t> > >&, PGLog::IndexedLog&, pg_missing_t&, > > std::__cxx11::basic_ostringstream<char, std::char_traits<char>, > > std::allocator<char> >&, DoutPrefixProvider const*, > > std::set<std::__cxx11::basic_string<char, std::char_traits<char>, > > std::allocator<char> >, std::less<std::__cxx11::basic_string<char, > > std::char_traits<char>, std::allocator<char> > >, > > std::allocator<std::__cxx11::basic_string<char, > > std::char_traits<char>, std::allocator<char> > > >*)+0x719) > > [0x5642d22e2fd9] > > 3: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x2f6) > > [0x5642d21172d6] > > 4: (OSD::load_pgs()+0x87d) [0x5642d205345d] > > 5: (OSD::init()+0x2026) [0x5642d205e7a6] > > 6: (main()+0x2ea5) [0x5642d1fd08f5] > > 7: (__libc_start_main()+0xf0) [0x7f728c77c830] > > 8: (_start()+0x29) [0x5642d2011f89] > > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |