For anyone that encounters this in the future, I was able to resolve the issue by finding the three osd's that the object is on. One by one I stop the osd, flushed the journal and used the objectstore tool to remove the data (sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-19 --journal-path /dev/disk/by-partlabel/journal19 --pool tier3-rbd-3X rbd_data.19cdf512ae8944a.000000000001bb56 remove). Then I started the osd again and let it recover before moving on to the next osd. After the object was deleted from all three osd's I ran a scrub on the PG (ceph pg scrub 3.f05). Once the scrub was finished the inconsistency went away. Note, the object in question was empty (size of zero bytes) before I started this process. I emptied the object by moving the rbd image to another pool. Rich On 24 October 2017 at 14:34, Richard Bade <hitrich@xxxxxxxxx> wrote: > What I'm thinking about trying is using the ceph-objectstore-tool to > remove the offending clone metadata. From the help the syntax is this: > ceph-objectstore-tool ... <object> remove-clone-metadata <cloneid> > i.e. something like for my object and expected clone from the log message > ceph-objectstore-tool rbd_data.19cdf512ae8944a.000000000001bb56 > remove-clone-metadata 148d2 > Anyone had experience with this? I'm not 100% sure if this will > resolve the issue or cause much the same situation (since it's already > expecting a clone that's not there currently). > > Rich > > On 21 October 2017 at 14:13, Brad Hubbard <bhubbard@xxxxxxxxxx> wrote: >> On Sat, Oct 21, 2017 at 1:59 AM, Richard Bade <hitrich@xxxxxxxxx> wrote: >>> Hi Lincoln, >>> Yes the object is 0-bytes on all OSD's. Has the same filesystem >>> date/time too. Before I removed the rbd image (migrated disk to >>> different pool) it was 4MB on all the OSD's and md5 checksum was the >>> same on all so it seems that only metadata is inconsistent. >>> Thanks for your suggestion, I just looked into this as I thought maybe >>> I can delete the object (since it's empty anyway). But I just get file >>> not found: >>> ~$ rados stat rbd_data.19cdf512ae8944a.000000000001bb56 --pool=tier3-rbd-3X >>> error stat-ing >>> tier3-rbd-3X/rbd_data.19cdf512ae8944a.000000000001bb56: (2) No such >>> file or directory >> >> Maybe try downing the osds involved? >> >>> >>> Regards, >>> Rich >>> >>> On 21 October 2017 at 04:32, Lincoln Bryant <lincolnb@xxxxxxxxxxxx> wrote: >>>> Hi Rich, >>>> >>>> Is the object inconsistent and 0-bytes on all OSDs? >>>> >>>> We ran into a similar issue on Jewel, where an object was empty across the board but had inconsistent metadata. Ultimately it was resolved by doing a "rados get" and then a "rados put" on the object. *However* that was a last ditch effort after I couldn't get any other repair option to work, and I have no idea if that will cause any issues down the road :) >>>> >>>> --Lincoln >>>> >>>>> On Oct 20, 2017, at 10:16 AM, Richard Bade <hitrich@xxxxxxxxx> wrote: >>>>> >>>>> Hi Everyone, >>>>> In our cluster running 0.94.10 we had a pg pop up as inconsistent >>>>> during scrub. Previously when this has happened running ceph pg repair >>>>> [pg_num] has resolved the problem. This time the repair runs but it >>>>> remains inconsistent. >>>>> ~$ ceph health detail >>>>> HEALTH_ERR 1 pgs inconsistent; 2 scrub errors; noout flag(s) set >>>>> pg 3.f05 is active+clean+inconsistent, acting [171,23,131] >>>>> 1 scrub errors >>>>> >>>>> The error in the logs is: >>>>> cstor01 ceph-mon: osd.171 10.233.202.21:6816/12694 45 : deep-scrub >>>>> 3.f05 3/68ab5f05/rbd_data.19cdf512ae8944a.000000000001bb56/snapdir >>>>> expected clone 3/68ab5f05/rbd_data.19cdf512ae8944a.000000000001bb56/148d2 >>>>> >>>>> Now, I've tried several things to resolve this. I've tried stopping >>>>> each of the osd's in turn and running a repair. I've located the rbd >>>>> image and removed it to empty out the object. The object is now zero >>>>> bytes but still inconsistent. I've tried stopping each osd, removing >>>>> the object and starting the osd again. It correctly identifies the >>>>> object as missing and repair works to fix this but it still remains >>>>> inconsistent. >>>>> I've run out of ideas. >>>>> The object is now zero bytes: >>>>> ~$ find /var/lib/ceph/osd/ceph-23/current/3.f05_head/ -name >>>>> "*19cdf512ae8944a.000000000001bb56*" -ls >>>>> 537598582 0 -rw-r--r-- 1 root root 0 Oct 21 >>>>> 03:54 /var/lib/ceph/osd/ceph-23/current/3.f05_head/DIR_5/DIR_0/DIR_F/DIR_5/DIR_B/rbd\\udata.19cdf512ae8944a.000000000001bb56__snapdir_68AB5F05__3 >>>>> >>>>> How can I resolve this? Is there some way to remove the empty object >>>>> completely? I saw reference to ceph-objectstore-tool which has some >>>>> options to remove-clone-metadata but I don't know how to use this. >>>>> Will using this to remove the mentioned 148d2 expected clone resolve >>>>> this? Or would this do the opposite as it would seem that it can't >>>>> find that clone? >>>>> Documentation on this tool is sparse. >>>>> >>>>> Any help here would be appreciated. >>>>> >>>>> Regards, >>>>> Rich >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@xxxxxxxxxxxxxx >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@xxxxxxxxxxxxxx >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> -- >> Cheers, >> Brad _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |