Hello Stefan, ceph-object-tool does not exist on my setup, do yo mean the command /usr/bin/ceph-objectstore-tool that is installed with the ceph-osd package ? I have the following situation here in Ceph Luminous: 2018-02-26 07:15:30.066393 7f0684acb700 -1 log_channel(cluster) log [ERR] : 5.111f shard 395 missing 5:f88e2b07:::rbd_data.8a09fb8793c74f.0000000000006dce:23152 2018-02-26 07:15:30.395189 7f0684acb700 -1 log_channel(cluster) log [ERR] : deep-scrub 5.111f 5:f88e2b07:::rbd_data.8a09fb8793c74f.0000000000006dce:23152 is an unexpected clone I did not understand how you actually fixed the problem. Could you provide more details ? thanks Saverio On 08.08.17 12:02, Stefan Priebe - Profihost AG wrote: > Hello Greg, > > Am 08.08.2017 um 11:56 schrieb Gregory Farnum: >> On Mon, Aug 7, 2017 at 11:55 PM Stefan Priebe - Profihost AG >> <s.priebe@xxxxxxxxxxxx <mailto:s.priebe@xxxxxxxxxxxx>> wrote: >> >> Hello, >> >> how can i fix this one: >> >> 2017-08-08 08:42:52.265321 osd.20 [ERR] repair 3.61a >> 3:58654d3d:::rbd_data.106dd406b8b4567.000000000000018c:9d455 is an >> unexpected clone >> 2017-08-08 08:43:04.914640 mon.0 [INF] HEALTH_ERR; 1 pgs inconsistent; 1 >> pgs repair; 1 scrub errors >> 2017-08-08 08:43:33.470246 osd.20 [ERR] 3.61a repair 1 errors, 0 fixed >> 2017-08-08 08:44:04.915148 mon.0 [INF] HEALTH_ERR; 1 pgs inconsistent; 1 >> scrub errors >> >> If i just delete manually the relevant files ceph is crashing. rados >> does not list those at all? >> >> How can i fix this? >> >> >> You've sent quite a few emails that have this story spread out, and I >> think you've tried several different steps to repair it that have been a >> bit difficult to track. >> >> It would be helpful if you could put the whole story in one place and >> explain very carefully exactly what you saw and how you responded. Stuff >> like manually copying around the wrong files, or files without a >> matching object info, could have done some very strange things. >> Also, basic debugging stuff like what version you're running will help. :) >> >> Also note that since you've said elsewhere you don't need this image, I >> don't think it's going to hurt you to leave it like this for a bit >> (though it will definitely mess up your monitoring). >> -Greg > > i'm sorry about that. You're correct. > > I was able to fix this just a few minutes ago by using the > ceph-object-tool and the remove operation to remove all left over files. > > I did this on all OSDs with the problematic pg. After that ceph was able > to fix itself. > > A better approach might be that ceph can recover itself from an > unexpected clone by just deleting it. > > Greets, > Stefan > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- SWITCH Saverio Proto, Peta Solutions Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland phone +41 44 268 15 15, direct +41 44 268 1573 saverio.proto@xxxxxxxxx, http://www.switch.ch http://www.switch.ch/stories _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |