Re: Help with inconsistent pg on EC pool, v9.0.2

[David Zafman <dzafman@xxxxxxxxxx>]

On 8/28/15 4:18 PM, Aaron Ten Clay wrote:
> How would I go about removing the bad PG with ceph-objectstore-tool? I'm
> having trouble finding any documentation for said tool.

ceph_objectsore_tool —data-path /var/lib/ceph/osd/ceph-0 —journal-path 
/var/lib/ceph/osd/ceph-0/journal —pgid 2.36s1 —op remove
> Is it safe to just move /var/lib/ceph/osd/ceph-21/current/2.36s1_head to
> another place and start the OSD process again?

yes
> Can I safely tar the directory with tar -cvp --xattrs -f
> /opt/osd-21-pg-2.36s1-removed_2015-08-28.tar
> /var/lib/ceph/osd/ceph-21/current/2.36s1_*, then rm -rf
> /var/lib/ceph/osd/ceph-21/current/2.36s1_*?

Better to use the --op export feature if you want to save the pg state:

ceph_objectsore_tool —data-path /var/lib/ceph/osd/ceph-X —journal-path 
/var/lib/ceph/osd/ceph-X/journal —pgid 2.36s1 —op export --file 
save2.36.s1.export
> Just want to make sure I don't do something silly and shoot myself in the
> foot.
>
> Thanks!
> -Aaron
>
> On Fri, Aug 28, 2015 at 12:16 PM, David Zafman <dzafman@xxxxxxxxxx> wrote:
>
> > I don't know about removing the OSD from the CRUSH map.  That seems like
> > overkill to me.
> >
> > I just realized a possible better way.  It would have been to take OSD
> > down not out.  Remove the ECs PG with the bad chunk.  Bring it up again and
> > let recovery repair just the single missing PG on the single OSD with no
> > other disruption.
> >
> > David
> >
> >
> > On 8/28/15 11:28 AM, Aaron Ten Clay wrote:
> >
> > > Thanks for the tip, David. I've marked osd.21 down and out and will wait
> > > for recovery. I've never had success manually manipulating the OSD
> > > contents
> > > - I assume I can achieve the same result by removing osd.21 from the CRUSH
> > > map, "ceph osd rm 21", then recreating it from scratch as though I'd lost
> > > a
> > > disk?
> > >
> > > -Aaron
> > >
> > > On Fri, Aug 28, 2015 at 11:17 AM, David Zafman <dzafman@xxxxxxxxxx>
> > > wrote:
> > >
> > > Without my latest branch which hasn't merged yet, you can't repair an EC
> > > > pg in the situation that the shard with a bad checksum is in the first k
> > > > chunks.
> > > >
> > > > A way to fix it would be to take that osd down/out and let recovery
> > > > regenerate the chunk.  Remove the pg from the osd (ceph-objectstore-tool)
> > > > and then you can bring the osd back up/in.
> > > >
> > > > David
> > > >
> > > >
> > > > On 8/28/15 11:06 AM, Samuel Just wrote:
> > > >
> > > > David, does this look familiar?
> > > > > -Sam
> > > > >
> > > > > On Fri, Aug 28, 2015 at 10:43 AM, Aaron Ten Clay <aarontc@xxxxxxxxxxx>
> > > > > wrote:
> > > > >
> > > > > Hi Cephers,
> > > > > > I'm trying to resolve an inconsistent pg on an erasure-coded pool,
> > > > > > running
> > > > > > Ceph 9.0.2. I can't seem to get Ceph to run a repair or even deep-scrub
> > > > > > the
> > > > > > pg again. Here's the background, with my attempted resolution steps
> > > > > > below.
> > > > > > Hopefully someone can steer me in the right direction. Thanks in
> > > > > > advance!
> > > > > >
> > > > > > Current state:
> > > > > > # ceph health detail
> > > > > > HEALTH_ERR 1 pgs inconsistent; 1 scrub errors; noout flag(s) set
> > > > > > pg 2.36 is active+clean+inconsistent, acting
> > > > > > [1,21,12,9,0,10,14,7,18,20,5,4,22,16]
> > > > > > 1 scrub errors
> > > > > > noout flag(s) set
> > > > > >
> > > > > > I started by looking at the log file for osd.1, where I found the cause
> > > > > > of
> > > > > > the inconsistent report:
> > > > > >
> > > > > > 2015-08-24 00:43:10.391621 7f09fcff9700  0 log_channel(cluster) log
> > > > > > [INF] :
> > > > > > 2.36 deep-scrub starts
> > > > > > 2015-08-24 01:54:59.933532 7f09fcff9700 -1 log_channel(cluster) log
> > > > > > [ERR] :
> > > > > > 2.36s0 shard 21(1): soid 576340b6/10000005990.00000199/head//2
> > > > > > candidate
> > > > > > had
> > > > > > a read error
> > > > > > 2015-08-24 02:34:41.380740 7f09fcff9700 -1 log_channel(cluster) log
> > > > > > [ERR] :
> > > > > > 2.36s0 deep-scrub 0 missing, 1 inconsistent objects
> > > > > > 2015-08-24 02:34:41.380757 7f09fcff9700 -1 log_channel(cluster) log
> > > > > > [ERR] :
> > > > > > 2.36 deep-scrub 1 errors
> > > > > >
> > > > > > I checked osd.21, where this report appears:
> > > > > >
> > > > > > 2015-08-24 01:54:56.477020 7f707cbd4700  0 osd.21 pg_epoch: 31958
> > > > > > pg[2.36s1(
> > > > > > v 31957'43013 (7132'39997,31957'43013] local-les=31951 n=34556 ec=136
> > > > > > les/c
> > > > > > 31951/31954 31945/31945/31924) [1,21,12,9,0,10,14,7,18,20,5,4,22,16]
> > > > > > r=1
> > > > > > lpr=31945 pi=1131-31944/7827 luod=0'0 crt=31957'43011 active]
> > > > > > _scan_list
> > > > > > 576340b6/10000005990.00000199/head//2 got incorrect hash on read
> > > > > >
> > > > > > So, based upon the ceph documentation, I thought I could repair the pg
> > > > > > by
> > > > > > executing "ceph pg repair 2.36". When I run this, while watching the
> > > > > > mon
> > > > > > log, I see the command dispatch:
> > > > > >
> > > > > > 2015-08-28 10:14:17.964017 mon.0 [INF] from='client.?
> > > > > > 10.42.5.61:0/1002181'
> > > > > > entity='client.admin' cmd=[{"prefix": "pg repair", "pgid": "2.36"}]:
> > > > > > dispatch
> > > > > >
> > > > > > But I never see a "finish" in the mon log, like most ceph commands
> > > > > > return.
> > > > > > (Not sure if I should expect to see a finish, just noting it doesn't
> > > > > > occur.)
> > > > > >
> > > > > > Also, tailing the logs for any OSD in the acting set for pg 2.36, I
> > > > > > never
> > > > > > see anything about a repair. The same case holds when I try "ceph pg
> > > > > > 2.36
> > > > > > deep-scrub" - command dispatched, but none of the OSDs care. In the
> > > > > > past
> > > > > > on
> > > > > > other clusters, I've seen "[INF] : pg.id repair starts" messages in
> > > > > > the
> > > > > > OSD
> > > > > > log after executing "ceph pg nn.yy repair".
> > > > > >
> > > > > > Further confusing me, I do see osd.1 start and finish other pg
> > > > > > deep-scrubs,
> > > > > > before and after executing "ceph pg 2.36 deep-scrub".
> > > > > >
> > > > > > I know EC pools are special in several ways, but nothing in the Ceph
> > > > > > manual
> > > > > > seems to indicate I can't deep-scrub or repair pgs in an EC pool...
> > > > > >
> > > > > > Thanks for reading and any suggestions. I'm happy to provide complete
> > > > > > log
> > > > > > files or more details if I've left out any information that could be
> > > > > > helpful.
> > > > > >
> > > > > > ceph -s: http://hastebin.com/xetohugibi
> > > > > > ceph pg dump: http://hastebin.com/bijehoheve
> > > > > > ceph -v: ceph version 9.0.2 (be422c8f5b494c77ebcf0f7b95e5d728ecacb7f0)
> > > > > > ceph osd dump: http://hastebin.com/fitajuzeca
> > > > > >
> > > > > > -Aaron
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > ceph-users mailing list
> > > > > > ceph-users@xxxxxxxxxxxxxx
> > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com