The feature bug for the tool is http://tracker.ceph.com/issues/12740. -Sam On Thu, Aug 20, 2015 at 2:52 PM, Samuel Just <sjust@xxxxxxxxxx> wrote: > Ah, this is kind of silly. I think you don't have 37 errors, but 2 > errors. pg 2.490 object > 3fac9490/rbd_data.eb5f22eb141f2.00000000000004ba/snapdir//2 is missing > snap 141. If you look at the objects after that in the log: > > 2015-08-20 20:15:44.865670 osd.19 10.12.2.6:6838/1861727 298 : cluster > [ERR] repair 2.490 > 68c89490/rbd_data.16796a3d1b58ba.0000000000000047/head//2 expected > clone 2d7b9490/rbd_data.18f92c3d1b58ba.0000000000006167/141//2 > 2015-08-20 20:15:44.865817 osd.19 10.12.2.6:6838/1861727 299 : cluster > [ERR] repair 2.490 > ded49490/rbd_data.11a25c7934d3d4.0000000000008a8a/head//2 expected > clone 68c89490/rbd_data.16796a3d1b58ba.0000000000000047/141//2 > > The clone from the second line matches the head object from the > previous line, and they have the same clone id. I *think* that the > first error is real, and the subsequent ones are just scrub being > dumb. Same deal with pg 2.c4. I just opened > http://tracker.ceph.com/issues/12738. > > The original problem is that > 3fac9490/rbd_data.eb5f22eb141f2.00000000000004ba/snapdir//2 and > 22ca30c4/rbd_data.e846e25a70bf7.0000000000000307/snapdir//2 are both > missing a clone. Not sure how that happened, my money is on a > cache/tiering evict racing with a snap trim. If you have any logging > or relevant information from when that happened, you should open a > bug. The 'snapdir' in the two object names indicates that the head > object has actually been deleted (which makes sense if you moved the > image to a new image and deleted the old one) and is only being kept > around since there are live snapshots. I suggest you leave the > snapshots for those images alone for the time being -- removing them > might cause the osd to crash trying to clean up the wierd on disk > state. Other than the leaked space from those two image snapshots and > the annoying spurious scrub errors, I think no actual corruption is > going on though. I created a tracker ticket for a feature that would > let ceph-objectstore-tool remove the spurious clone from the > head/snapdir metadata. > > Am I right that you haven't actually seen any osd crashes or user > visible corruption (except possibly on snapshots of those two images)? > -Sam > > On Thu, Aug 20, 2015 at 10:07 AM, Voloshanenko Igor > <igor.voloshanenko@xxxxxxxxx> wrote: >> Inktank: >> https://download.inktank.com/docs/ICE%201.2%20-%20Cache%20and%20Erasure%20Coding%20FAQ.pdf >> >> Mail-list: >> https://www.mail-archive.com/ceph-users@xxxxxxxxxxxxxx/msg18338.html >> >> 2015-08-20 20:06 GMT+03:00 Samuel Just <sjust@xxxxxxxxxx>: >>> >>> Which docs? >>> -Sam >>> >>> On Thu, Aug 20, 2015 at 9:57 AM, Voloshanenko Igor >>> <igor.voloshanenko@xxxxxxxxx> wrote: >>> > Not yet. I will create. >>> > But according to mail lists and Inktank docs - it's expected behaviour >>> > when >>> > cache enable >>> > >>> > 2015-08-20 19:56 GMT+03:00 Samuel Just <sjust@xxxxxxxxxx>: >>> >> >>> >> Is there a bug for this in the tracker? >>> >> -Sam >>> >> >>> >> On Thu, Aug 20, 2015 at 9:54 AM, Voloshanenko Igor >>> >> <igor.voloshanenko@xxxxxxxxx> wrote: >>> >> > Issue, that in forward mode, fstrim doesn't work proper, and when we >>> >> > take >>> >> > snapshot - data not proper update in cache layer, and client (ceph) >>> >> > see >>> >> > damaged snap.. As headers requested from cache layer. >>> >> > >>> >> > 2015-08-20 19:53 GMT+03:00 Samuel Just <sjust@xxxxxxxxxx>: >>> >> >> >>> >> >> What was the issue? >>> >> >> -Sam >>> >> >> >>> >> >> On Thu, Aug 20, 2015 at 9:41 AM, Voloshanenko Igor >>> >> >> <igor.voloshanenko@xxxxxxxxx> wrote: >>> >> >> > Samuel, we turned off cache layer few hours ago... >>> >> >> > I will post ceph.log in few minutes >>> >> >> > >>> >> >> > For snap - we found issue, was connected with cache tier.. >>> >> >> > >>> >> >> > 2015-08-20 19:23 GMT+03:00 Samuel Just <sjust@xxxxxxxxxx>: >>> >> >> >> >>> >> >> >> Ok, you appear to be using a replicated cache tier in front of a >>> >> >> >> replicated base tier. Please scrub both inconsistent pgs and >>> >> >> >> post >>> >> >> >> the >>> >> >> >> ceph.log from before when you started the scrub until after. >>> >> >> >> Also, >>> >> >> >> what command are you using to take snapshots? >>> >> >> >> -Sam >>> >> >> >> >>> >> >> >> On Thu, Aug 20, 2015 at 3:59 AM, Voloshanenko Igor >>> >> >> >> <igor.voloshanenko@xxxxxxxxx> wrote: >>> >> >> >> > Hi Samuel, we try to fix it in trick way. >>> >> >> >> > >>> >> >> >> > we check all rbd_data chunks from logs (OSD) which are >>> >> >> >> > affected, >>> >> >> >> > then >>> >> >> >> > query >>> >> >> >> > rbd info to compare which rbd consist bad rbd_data, after that >>> >> >> >> > we >>> >> >> >> > mount >>> >> >> >> > this >>> >> >> >> > rbd as rbd0, create empty rbd, and DD all info from bad volume >>> >> >> >> > to >>> >> >> >> > new >>> >> >> >> > one. >>> >> >> >> > >>> >> >> >> > But after that - scrub errors growing... Was 15 errors.. .Now >>> >> >> >> > 35... >>> >> >> >> > We >>> >> >> >> > laos >>> >> >> >> > try to out OSD which was lead, but after rebalancing this 2 pgs >>> >> >> >> > still >>> >> >> >> > have >>> >> >> >> > 35 scrub errors... >>> >> >> >> > >>> >> >> >> > ceph osd getmap -o <outfile> - attached >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > 2015-08-18 18:48 GMT+03:00 Samuel Just <sjust@xxxxxxxxxx>: >>> >> >> >> >> >>> >> >> >> >> Is the number of inconsistent objects growing? Can you attach >>> >> >> >> >> the >>> >> >> >> >> whole ceph.log from the 6 hours before and after the snippet >>> >> >> >> >> you >>> >> >> >> >> linked above? Are you using cache/tiering? Can you attach >>> >> >> >> >> the >>> >> >> >> >> osdmap >>> >> >> >> >> (ceph osd getmap -o <outfile>)? >>> >> >> >> >> -Sam >>> >> >> >> >> >>> >> >> >> >> On Tue, Aug 18, 2015 at 4:15 AM, Voloshanenko Igor >>> >> >> >> >> <igor.voloshanenko@xxxxxxxxx> wrote: >>> >> >> >> >> > ceph - 0.94.2 >>> >> >> >> >> > Its happen during rebalancing >>> >> >> >> >> > >>> >> >> >> >> > I thought too, that some OSD miss copy, but looks like all >>> >> >> >> >> > miss... >>> >> >> >> >> > So any advice in which direction i need to go >>> >> >> >> >> > >>> >> >> >> >> > 2015-08-18 14:14 GMT+03:00 Gregory Farnum >>> >> >> >> >> > <gfarnum@xxxxxxxxxx>: >>> >> >> >> >> >> >>> >> >> >> >> >> From a quick peek it looks like some of the OSDs are >>> >> >> >> >> >> missing >>> >> >> >> >> >> clones >>> >> >> >> >> >> of >>> >> >> >> >> >> objects. I'm not sure how that could happen and I'd expect >>> >> >> >> >> >> the >>> >> >> >> >> >> pg >>> >> >> >> >> >> repair to handle that but if it's not there's probably >>> >> >> >> >> >> something >>> >> >> >> >> >> wrong; what version of Ceph are you running? Sam, is this >>> >> >> >> >> >> something >>> >> >> >> >> >> you've seen, a new bug, or some kind of config issue? >>> >> >> >> >> >> -Greg >>> >> >> >> >> >> >>> >> >> >> >> >> On Tue, Aug 18, 2015 at 6:27 AM, Voloshanenko Igor >>> >> >> >> >> >> <igor.voloshanenko@xxxxxxxxx> wrote: >>> >> >> >> >> >> > Hi all, at our production cluster, due high rebalancing >>> >> >> >> >> >> > ((( >>> >> >> >> >> >> > we >>> >> >> >> >> >> > have 2 >>> >> >> >> >> >> > pgs in >>> >> >> >> >> >> > inconsistent state... >>> >> >> >> >> >> > >>> >> >> >> >> >> > root@temp:~# ceph health detail | grep inc >>> >> >> >> >> >> > HEALTH_ERR 2 pgs inconsistent; 18 scrub errors >>> >> >> >> >> >> > pg 2.490 is active+clean+inconsistent, acting [56,15,29] >>> >> >> >> >> >> > pg 2.c4 is active+clean+inconsistent, acting [56,10,42] >>> >> >> >> >> >> > >>> >> >> >> >> >> > From OSD logs, after recovery attempt: >>> >> >> >> >> >> > >>> >> >> >> >> >> > root@test:~# ceph pg dump | grep -i incons | cut -f 1 | >>> >> >> >> >> >> > while >>> >> >> >> >> >> > read >>> >> >> >> >> >> > i; >>> >> >> >> >> >> > do >>> >> >> >> >> >> > ceph pg repair ${i} ; done >>> >> >> >> >> >> > dumped all in format plain >>> >> >> >> >> >> > instructing pg 2.490 on osd.56 to repair >>> >> >> >> >> >> > instructing pg 2.c4 on osd.56 to repair >>> >> >> >> >> >> > >>> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:51:2015-08-18 >>> >> >> >> >> >> > 07:26:37.035910 >>> >> >> >> >> >> > 7f94663b3700 >>> >> >> >> >> >> > -1 >>> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub 2.490 >>> >> >> >> >> >> > f5759490/rbd_data.1631755377d7e.00000000000004da/head//2 >>> >> >> >> >> >> > expected >>> >> >> >> >> >> > clone >>> >> >> >> >> >> > 90c59490/rbd_data.eb486436f2beb.0000000000007a65/141//2 >>> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:52:2015-08-18 >>> >> >> >> >> >> > 07:26:37.035960 >>> >> >> >> >> >> > 7f94663b3700 >>> >> >> >> >> >> > -1 >>> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub 2.490 >>> >> >> >> >> >> > fee49490/rbd_data.12483d3ba0794b.000000000000522f/head//2 >>> >> >> >> >> >> > expected >>> >> >> >> >> >> > clone >>> >> >> >> >> >> > f5759490/rbd_data.1631755377d7e.00000000000004da/141//2 >>> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:53:2015-08-18 >>> >> >> >> >> >> > 07:26:37.036133 >>> >> >> >> >> >> > 7f94663b3700 >>> >> >> >> >> >> > -1 >>> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub 2.490 >>> >> >> >> >> >> > a9b39490/rbd_data.12483d3ba0794b.00000000000037b3/head//2 >>> >> >> >> >> >> > expected >>> >> >> >> >> >> > clone >>> >> >> >> >> >> > fee49490/rbd_data.12483d3ba0794b.000000000000522f/141//2 >>> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:54:2015-08-18 >>> >> >> >> >> >> > 07:26:37.036243 >>> >> >> >> >> >> > 7f94663b3700 >>> >> >> >> >> >> > -1 >>> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub 2.490 >>> >> >> >> >> >> > bac19490/rbd_data.1238e82ae8944a.000000000000032e/head//2 >>> >> >> >> >> >> > expected >>> >> >> >> >> >> > clone >>> >> >> >> >> >> > a9b39490/rbd_data.12483d3ba0794b.00000000000037b3/141//2 >>> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:55:2015-08-18 >>> >> >> >> >> >> > 07:26:37.036289 >>> >> >> >> >> >> > 7f94663b3700 >>> >> >> >> >> >> > -1 >>> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub 2.490 >>> >> >> >> >> >> > 98519490/rbd_data.123e9c2ae8944a.0000000000000807/head//2 >>> >> >> >> >> >> > expected >>> >> >> >> >> >> > clone >>> >> >> >> >> >> > bac19490/rbd_data.1238e82ae8944a.000000000000032e/141//2 >>> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:56:2015-08-18 >>> >> >> >> >> >> > 07:26:37.036314 >>> >> >> >> >> >> > 7f94663b3700 >>> >> >> >> >> >> > -1 >>> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub 2.490 >>> >> >> >> >> >> > c3c09490/rbd_data.1238e82ae8944a.0000000000000c2b/head//2 >>> >> >> >> >> >> > expected >>> >> >> >> >> >> > clone >>> >> >> >> >> >> > 98519490/rbd_data.123e9c2ae8944a.0000000000000807/141//2 >>> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:57:2015-08-18 >>> >> >> >> >> >> > 07:26:37.036363 >>> >> >> >> >> >> > 7f94663b3700 >>> >> >> >> >> >> > -1 >>> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub 2.490 >>> >> >> >> >> >> > 28809490/rbd_data.edea7460fe42b.00000000000001d9/head//2 >>> >> >> >> >> >> > expected >>> >> >> >> >> >> > clone >>> >> >> >> >> >> > c3c09490/rbd_data.1238e82ae8944a.0000000000000c2b/141//2 >>> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:58:2015-08-18 >>> >> >> >> >> >> > 07:26:37.036432 >>> >> >> >> >> >> > 7f94663b3700 >>> >> >> >> >> >> > -1 >>> >> >> >> >> >> > log_channel(cluster) log [ERR] : deep-scrub 2.490 >>> >> >> >> >> >> > e1509490/rbd_data.1423897545e146.00000000000009a6/head//2 >>> >> >> >> >> >> > expected >>> >> >> >> >> >> > clone >>> >> >> >> >> >> > 28809490/rbd_data.edea7460fe42b.00000000000001d9/141//2 >>> >> >> >> >> >> > /var/log/ceph/ceph-osd.56.log:59:2015-08-18 >>> >> >> >> >> >> > 07:26:38.548765 >>> >> >> >> >> >> > 7f94663b3700 >>> >> >> >> >> >> > -1 >>> >> >> >> >> >> > log_channel(cluster) log [ERR] : 2.490 deep-scrub 17 >>> >> >> >> >> >> > errors >>> >> >> >> >> >> > >>> >> >> >> >> >> > So, how i can solve "expected clone" situation by hand? >>> >> >> >> >> >> > Thank in advance! >>> >> >> >> >> >> > >>> >> >> >> >> >> > >>> >> >> >> >> >> > >>> >> >> >> >> >> > _______________________________________________ >>> >> >> >> >> >> > ceph-users mailing list >>> >> >> >> >> >> > ceph-users@xxxxxxxxxxxxxx >>> >> >> >> >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> > >>> >> >> > >>> >> > >>> >> > >>> > >>> > >> >> _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |