Am 26.02.2018 um 09:54 schrieb Saverio Proto: > Hello Stefan, > > ceph-object-tool does not exist on my setup, do yo mean the command > /usr/bin/ceph-objectstore-tool that is installed with the ceph-osd package ? Yes sorry i meant the ceph-objectstore-tool tool. With that you can remove objects. > > I have the following situation here in Ceph Luminous: > > 2018-02-26 07:15:30.066393 7f0684acb700 -1 log_channel(cluster) log > [ERR] : 5.111f shard 395 missing > 5:f88e2b07:::rbd_data.8a09fb8793c74f.0000000000006dce:23152 > 2018-02-26 07:15:30.395189 7f0684acb700 -1 log_channel(cluster) log > [ERR] : deep-scrub 5.111f > 5:f88e2b07:::rbd_data.8a09fb8793c74f.0000000000006dce:23152 is an > unexpected clone > > I did not understand how you actually fixed the problem. Could you > provide more details ? something like: ceph-objectstore-tool --data-path /.../osd.$OSD/ --journal-path /dev/disk/by-partlabel/journal$OSD rbd_data.$RBD remove-clone-metadata $CLONEID > > thanks > > Saverio > > > On 08.08.17 12:02, Stefan Priebe - Profihost AG wrote: >> Hello Greg, >> >> Am 08.08.2017 um 11:56 schrieb Gregory Farnum: >>> On Mon, Aug 7, 2017 at 11:55 PM Stefan Priebe - Profihost AG >>> <s.priebe@xxxxxxxxxxxx <mailto:s.priebe@xxxxxxxxxxxx>> wrote: >>> >>> Hello, >>> >>> how can i fix this one: >>> >>> 2017-08-08 08:42:52.265321 osd.20 [ERR] repair 3.61a >>> 3:58654d3d:::rbd_data.106dd406b8b4567.000000000000018c:9d455 is an >>> unexpected clone >>> 2017-08-08 08:43:04.914640 mon.0 [INF] HEALTH_ERR; 1 pgs inconsistent; 1 >>> pgs repair; 1 scrub errors >>> 2017-08-08 08:43:33.470246 osd.20 [ERR] 3.61a repair 1 errors, 0 fixed >>> 2017-08-08 08:44:04.915148 mon.0 [INF] HEALTH_ERR; 1 pgs inconsistent; 1 >>> scrub errors >>> >>> If i just delete manually the relevant files ceph is crashing. rados >>> does not list those at all? >>> >>> How can i fix this? >>> >>> >>> You've sent quite a few emails that have this story spread out, and I >>> think you've tried several different steps to repair it that have been a >>> bit difficult to track. >>> >>> It would be helpful if you could put the whole story in one place and >>> explain very carefully exactly what you saw and how you responded. Stuff >>> like manually copying around the wrong files, or files without a >>> matching object info, could have done some very strange things. >>> Also, basic debugging stuff like what version you're running will help. :) >>> >>> Also note that since you've said elsewhere you don't need this image, I >>> don't think it's going to hurt you to leave it like this for a bit >>> (though it will definitely mess up your monitoring). >>> -Greg >> >> i'm sorry about that. You're correct. >> >> I was able to fix this just a few minutes ago by using the >> ceph-object-tool and the remove operation to remove all left over files. >> >> I did this on all OSDs with the problematic pg. After that ceph was able >> to fix itself. >> >> A better approach might be that ceph can recover itself from an >> unexpected clone by just deleting it. >> >> Greets, >> Stefan >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |