Re: All shards of PG missing object and inconsistent

[Gregory Farnum <gfarnum@xxxxxxxxxx>]

On Fri, Sep 21, 2018 at 5:04 PM Thomas White <thomas@xxxxxxxxxxxxxx> wrote:

Hi all,

 

I have recently performed a few tasks, namely purging several buckets from our RGWs and added additional hosts into Ceph causing some data movement for a rebalance. As this is now almost completed, I kicked off some deep scrubs and one PG is now returning the following information:

 

2018-09-21 23:17:59.717286 7f2f16796700 -1 log_channel(cluster) log [ERR] : 14.1b18 shard 313 missing 14:18daa344:::default.162489536.28__shadow_24TB%2f24TB%2fDESIGNTEAM%2fPROJECTS%2fDLA Piper%2f_DLA_4033_ Global Thought Leadership%2fFilm%2f04 Assets%2fFootage %ef%80%a2 Audio Sync%2fDLA_Thought_Leadership_NYC%2fCam_1%2fA13I1483.MOV.2~5v3nJDNLONBYszy54ZXZZQgos1D4Ywp.359_6:head

2018-09-21 23:17:59.717292 7f2f16796700 -1 log_channel(cluster) log [ERR] : 14.1b18 shard 665 missing 14:18daa344:::default.162489536.28__shadow_24TB%2f24TB%2fDESIGNTEAM%2fPROJECTS%2fDLA Piper%2f_DLA_4033_ Global Thought Leadership%2fFilm%2f04 Assets%2fFootage %ef%80%a2 Audio Sync%2fDLA_Thought_Leadership_NYC%2fCam_1%2fA13I1483.MOV.2~5v3nJDNLONBYszy54ZXZZQgos1D4Ywp.359_6:head

2018-09-21 23:17:59.885884 7f2f16796700 -1 log_channel(cluster) log [ERR] : 14.1b18 shard 385 missing 14:18daa344:::default.162489536.28__shadow_24TB%2f24TB%2fDESIGNTEAM%2fPROJECTS%2fDLA Piper%2f_DLA_4033_ Global Thought Leadership%2fFilm%2f04 Assets%2fFootage %ef%80%a2 Audio Sync%2fDLA_Thought_Leadership_NYC%2fCam_1%2fA13I1483.MOV.2~5v3nJDNLONBYszy54ZXZZQgos1D4Ywp.359_6:head

2018-09-21 23:20:24.954402 7f2f16796700 -1 log_channel(cluster) log [ERR] : 14.1b18 scrub stat mismatch, got 44026/44025 objects, 0/0 clones, 44026/44025 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 45423386817/45419192513 bytes, 0/0 hit_set_archive bytes.

2018-09-21 23:20:24.954418 7f2f16796700 -1 log_channel(cluster) log [ERR] : 14.1b18 scrub 1 missing, 0 inconsistent objects

2018-09-21 23:20:24.954421 7f2f16796700 -1 log_channel(cluster) log [ERR] : 14.1b18 scrub 4 errors

 

The object I recognise by name as belonging to a bucket purged earlier in the day, and is meant to be deleted.

That's distressing. Has anything untoward happened to the cluster? Have you tried re-scrubbing the PG and seeing if it continues to believe that object is missing?

 

What would be the best means to resolve this inconsistency when the object is supposed to be absent?

One easy way to is to write the object (with nonsense) and then delete it.

-Greg

 

 

Kind Regards,

 

Thomas

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com