Re: Ceph strange issue after adding a cache OSD.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I take it you have size =2 or min_size=1 or something like that for the cache pool? 1 OSD shouldn’t prevent PG's from recovering.

Your best bet would be to see if the PG that is causing the assert can be removed and let the OSD start up. If you are lucky, the PG causing the problems might not be one which also has unfound objects, otherwise you are likely have to get heavily involved in recovering objects with the object store tool.

> -----Original Message-----
> From: Daznis [mailto:daznis@xxxxxxxxx]
> Sent: 23 November 2016 13:56
> To: Nick Fisk <nick@xxxxxxxxxx>
> Cc: ceph-users <ceph-users@xxxxxxxxxxxxxx>
> Subject: Re:  Ceph strange issue after adding a cache OSD.
> 
> No, it's still missing some PGs and objects and can't recover as it's blocked by that OSD. I can boot the OSD up by removing all the PG
> related files from current directory, but that doesn't solve the missing objects problem. Not really sure if I can move the object back to
> their place manually, but I will try it.
> 
> On Wed, Nov 23, 2016 at 3:08 PM, Nick Fisk <nick@xxxxxxxxxx> wrote:
> > Sorry, I'm afraid I'm out of ideas about that one, that error doesn't mean very much to me. The code suggests the OSD is trying to
> get an attr from the disk/filesystem, but for some reason it doesn't like that. You could maybe whack the debug logging for OSD and
> filestore up to max and try and see what PG/file is accessed just before the crash, but I'm not sure what the fix would be, even if you
> manage to locate the dodgy PG.
> >
> > Does the cluster have all PG's recovered now? Unless anyone else can comment, you might be best removing/wiping and then re-
> adding the OSD.
> >
> >> -----Original Message-----
> >> From: Daznis [mailto:daznis@xxxxxxxxx]
> >> Sent: 23 November 2016 12:55
> >> To: Nick Fisk <nick@xxxxxxxxxx>
> >> Cc: ceph-users <ceph-users@xxxxxxxxxxxxxx>
> >> Subject: Re:  Ceph strange issue after adding a cache OSD.
> >>
> >> Thank you. That helped quite a lot. Now I'm just stuck with one OSD crashing with:
> >>
> >> osd/PG.cc: In function 'static int PG::peek_map_epoch(ObjectStore*,
> >> spg_t, epoch_t*, ceph::bufferlist*)' thread 7f36bbdd6880 time
> >> 2016-11-23 13:42:43.27
> >> 8539
> >> osd/PG.cc: 2911: FAILED assert(r > 0)
> >>
> >>  ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90)
> >>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> >> const*)+0x85) [0xbde2c5]
> >>  2: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*,
> >> ceph::buffer::list*)+0x8ba) [0x7cf4da]
> >>  3: (OSD::load_pgs()+0x9ef) [0x6bd31f]
> >>  4: (OSD::init()+0x181a) [0x6c0e8a]
> >>  5: (main()+0x29dd) [0x6484bd]
> >>  6: (__libc_start_main()+0xf5) [0x7f36b916bb15]
> >>  7: /usr/bin/ceph-osd() [0x661ea9]
> >>
> >> On Wed, Nov 23, 2016 at 12:31 PM, Nick Fisk <nick@xxxxxxxxxx> wrote:
> >> >> -----Original Message-----
> >> >> From: Daznis [mailto:daznis@xxxxxxxxx]
> >> >> Sent: 23 November 2016 10:17
> >> >> To: nick@xxxxxxxxxx
> >> >> Cc: ceph-users <ceph-users@xxxxxxxxxxxxxx>
> >> >> Subject: Re:  Ceph strange issue after adding a cache OSD.
> >> >>
> >> >> Hi,
> >> >>
> >> >>
> >> >> Looks like one of my colleagues increased the PG number before it
> >> >> finished. I was flushing the whole cache tier and it's currently
> >> >> stuck on ~80 GB of data, because of the OSD crashes. I will look
> >> >> into the hitset counts and check what can be done. Will provide an
> >> update if I find anything or fix the issue.
> >> >
> >> > So I'm guessing when the PG split, the stats/hit_sets are not how
> >> > the OSD is expecting them to be and causes the crash. I would
> >> expect this has been caused by the PG splitting rather than
> >> introducing extra OSD's. If you manage to get things stable by
> >> bumping up the hitset count, then you probably want to try and do a scrub to try and clean up the stats, which may then stop this
> happening when the hitset comes round to being trimmed again.
> >> >
> >> >>
> >> >>
> >> >> On Wed, Nov 23, 2016 at 12:04 PM, Nick Fisk <nick@xxxxxxxxxx> wrote:
> >> >> > Hi Daznis,
> >> >> >
> >> >> > I'm not sure how much help I can be, but I will try my best.
> >> >> >
> >> >> > I think the post-split stats error is probably benign, although
> >> >> > I think this suggests you also increased the number of PG's in
> >> >> > your cache pool? If so did you do this before or after you added
> >> >> > the
> >> >> extra OSD's?  This may have been the cause.
> >> >> >
> >> >> > On to the actual assert, this looks like it's part of the code
> >> >> > which trims the tiering hit set's. I don't understand why its
> >> >> > crashing out, but it must be related to an invalid or missing
> >> >> > hitset I would
> >> >> imagine.
> >> >> >
> >> >> > https://github.com/ceph/ceph/blob/v0.94.9/src/osd/ReplicatedPG.c
> >> >> > c#L
> >> >> > 104
> >> >> > 85
> >> >> >
> >> >> > The only thing I could think of from looking at in the code is
> >> >> > that the function loops through all hitsets that are above the
> >> >> > max number (hit_set_count). I wonder if setting this number
> >> >> > higher would
> >> >> mean it won't try and trim any hitsets and let things recover?
> >> >> >
> >> >> > DISCLAIMER
> >> >> > This is a hunch, it might not work or could possibly even make
> >> >> > things worse. Otherwise wait for someone who has a better idea to comment.
> >> >> >
> >> >> > Nick
> >> >> >
> >> >> >
> >> >> >
> >> >> >> -----Original Message-----
> >> >> >> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On
> >> >> >> Behalf Of Daznis
> >> >> >> Sent: 23 November 2016 05:57
> >> >> >> To: ceph-users <ceph-users@xxxxxxxxxxxxxx>
> >> >> >> Subject:  Ceph strange issue after adding a cache OSD.
> >> >> >>
> >> >> >> Hello,
> >> >> >>
> >> >> >>
> >> >> >> The story goes like this.
> >> >> >> I have added another 3 drives to the caching layer. OSDs were
> >> >> >> added to crush map one by one after each successful rebalance.
> >> >> >> When
> >> >> > I
> >> >> >> added the last OSD and went away for about an hour I noticed
> >> >> >> that it's still not finished rebalancing. Further investigation
> >> >> > showed me
> >> >> >> that it one of the older cache SSD was restarting like crazy
> >> >> >> before full boot. So I shut it down and waited for a rebalance
> >> >> > without that
> >> >> >> OSD. Less than an hour later I had another 2 OSD restarting
> >> >> >> like crazy. I tried running scrubs on the PG's logs asked me
> >> >> >> to, but
> >> >> > that did
> >> >> >> not help. I'm currently stuck with " 8 scrub errors" and a complete dead cluster.
> >> >> >>
> >> >> >> log_channel(cluster) log [WRN] : pg 15.8d has invalid
> >> >> >> (post-split) stats; must scrub before tier agent can activate
> >> >> >>
> >> >> >>
> >> >> >> I need help with OSD from crashing. Crash log:
> >> >> >>      0> 2016-11-23 06:41:43.365602 7f935b4eb700 -1
> >> >> >> osd/ReplicatedPG.cc: In function 'void
> >> >> >> ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*, unsigned int)'
> >> >> >> thread 7f935b4eb700 time 2016-11-23 06:41:43.363067
> >> >> >> osd/ReplicatedPG.cc: 10521: FAILED assert(obc)
> >> >> >>
> >> >> >>  ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90)
> >> >> >>  1: (ceph::__ceph_assert_fail(char const*, char const*, int,
> >> >> >> char
> >> >> >> const*)+0x85) [0xbde2c5]
> >> >> >>  2: (ReplicatedPG::hit_set_trim(ReplicatedPG::RepGather*,
> >> >> >> unsigned
> >> >> >> int)+0x75f) [0x87e89f]
> >> >> >>  3: (ReplicatedPG::hit_set_persist()+0xedb) [0x87f8bb]
> >> >> >>  4:
> >> >> >> (ReplicatedPG::do_op(std::tr1::shared_ptr<OpRequest>&)+0xe3a)
> >> >> >> [0x8a11aa]
> >> >> >>  5: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&,
> >> >> >> ThreadPool::TPHandle&)+0x68a) [0x83c37a]
> >> >> >>  6: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
> >> >> >> std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x405)
> >> >> >> [0x69af05]
> >> >> >>  7: (OSD::ShardedOpWQ::_process(unsigned int,
> >> >> >> ceph::heartbeat_handle_d*)+0x333) [0x69b473]
> >> >> >>  8: (ShardedThreadPool::shardedthreadpool_worker(unsigned
> >> >> >> int)+0x86f) [0xbcd9cf]
> >> >> >>  9: (ShardedThreadPool::WorkThreadSharded::entry()+0x10)
> >> >> >> [0xbcfb00]
> >> >> >>  10: (()+0x7dc5) [0x7f93b9df4dc5]
> >> >> >>  11: (clone()+0x6d) [0x7f93b88d5ced]
> >> >> >>  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
> >> >> >>
> >> >> >>
> >> >> >> I have tried looking with  full debug enabled, but those logs
> >> >> >> didn't help me much. I have tried to evict the cache layer, but
> >> >> >> some objects are stuck and can't be removed. Any suggestions
> >> >> >> would be
> >> >> greatly appreciated.
> >> >> >> _______________________________________________
> >> >> >> ceph-users mailing list
> >> >> >> ceph-users@xxxxxxxxxxxxxx
> >> >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >> >
> >> >
> >

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux