What I'm thinking about trying is using the ceph-objectstore-tool to remove the offending clone metadata. From the help the syntax is this: ceph-objectstore-tool ... <object> remove-clone-metadata <cloneid> i.e. something like for my object and expected clone from the log message ceph-objectstore-tool rbd_data.19cdf512ae8944a.000000000001bb56 remove-clone-metadata 148d2 Anyone had experience with this? I'm not 100% sure if this will resolve the issue or cause much the same situation (since it's already expecting a clone that's not there currently). Rich On 21 October 2017 at 14:13, Brad Hubbard <bhubbard@xxxxxxxxxx> wrote: > On Sat, Oct 21, 2017 at 1:59 AM, Richard Bade <hitrich@xxxxxxxxx> wrote: >> Hi Lincoln, >> Yes the object is 0-bytes on all OSD's. Has the same filesystem >> date/time too. Before I removed the rbd image (migrated disk to >> different pool) it was 4MB on all the OSD's and md5 checksum was the >> same on all so it seems that only metadata is inconsistent. >> Thanks for your suggestion, I just looked into this as I thought maybe >> I can delete the object (since it's empty anyway). But I just get file >> not found: >> ~$ rados stat rbd_data.19cdf512ae8944a.000000000001bb56 --pool=tier3-rbd-3X >> error stat-ing >> tier3-rbd-3X/rbd_data.19cdf512ae8944a.000000000001bb56: (2) No such >> file or directory > > Maybe try downing the osds involved? > >> >> Regards, >> Rich >> >> On 21 October 2017 at 04:32, Lincoln Bryant <lincolnb@xxxxxxxxxxxx> wrote: >>> Hi Rich, >>> >>> Is the object inconsistent and 0-bytes on all OSDs? >>> >>> We ran into a similar issue on Jewel, where an object was empty across the board but had inconsistent metadata. Ultimately it was resolved by doing a "rados get" and then a "rados put" on the object. *However* that was a last ditch effort after I couldn't get any other repair option to work, and I have no idea if that will cause any issues down the road :) >>> >>> --Lincoln >>> >>>> On Oct 20, 2017, at 10:16 AM, Richard Bade <hitrich@xxxxxxxxx> wrote: >>>> >>>> Hi Everyone, >>>> In our cluster running 0.94.10 we had a pg pop up as inconsistent >>>> during scrub. Previously when this has happened running ceph pg repair >>>> [pg_num] has resolved the problem. This time the repair runs but it >>>> remains inconsistent. >>>> ~$ ceph health detail >>>> HEALTH_ERR 1 pgs inconsistent; 2 scrub errors; noout flag(s) set >>>> pg 3.f05 is active+clean+inconsistent, acting [171,23,131] >>>> 1 scrub errors >>>> >>>> The error in the logs is: >>>> cstor01 ceph-mon: osd.171 10.233.202.21:6816/12694 45 : deep-scrub >>>> 3.f05 3/68ab5f05/rbd_data.19cdf512ae8944a.000000000001bb56/snapdir >>>> expected clone 3/68ab5f05/rbd_data.19cdf512ae8944a.000000000001bb56/148d2 >>>> >>>> Now, I've tried several things to resolve this. I've tried stopping >>>> each of the osd's in turn and running a repair. I've located the rbd >>>> image and removed it to empty out the object. The object is now zero >>>> bytes but still inconsistent. I've tried stopping each osd, removing >>>> the object and starting the osd again. It correctly identifies the >>>> object as missing and repair works to fix this but it still remains >>>> inconsistent. >>>> I've run out of ideas. >>>> The object is now zero bytes: >>>> ~$ find /var/lib/ceph/osd/ceph-23/current/3.f05_head/ -name >>>> "*19cdf512ae8944a.000000000001bb56*" -ls >>>> 537598582 0 -rw-r--r-- 1 root root 0 Oct 21 >>>> 03:54 /var/lib/ceph/osd/ceph-23/current/3.f05_head/DIR_5/DIR_0/DIR_F/DIR_5/DIR_B/rbd\\udata.19cdf512ae8944a.000000000001bb56__snapdir_68AB5F05__3 >>>> >>>> How can I resolve this? Is there some way to remove the empty object >>>> completely? I saw reference to ceph-objectstore-tool which has some >>>> options to remove-clone-metadata but I don't know how to use this. >>>> Will using this to remove the mentioned 148d2 expected clone resolve >>>> this? Or would this do the opposite as it would seem that it can't >>>> find that clone? >>>> Documentation on this tool is sparse. >>>> >>>> Any help here would be appreciated. >>>> >>>> Regards, >>>> Rich >>>> _______________________________________________ >>>> ceph-users mailing list >>>> ceph-users@xxxxxxxxxxxxxx >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > Cheers, > Brad _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |