Thanks, we are currently scanning our object storage. It looks like we can detect the missing objects that return “No Such Key” looking at all “__multipart_” objects returned by radosgw-admin bucket radoslist, and checking if they exist using rados stat. We are currently not looking at shadow objects as our approach already yields more instances of this problem. > On 19 Nov 2020, at 09:09, Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx> wrote: > >> - The head object had a size of 0. >> - There was an object with a ’shadow’ in its name, belonging to that path. > That is normal. What is not normal is if there are NO shadow objects. > > On 18/11/2020 10:06, Denis Krienbühl wrote: >> It looks like a single-part object. But we did replace that object last night from backup, so I can’t know for sure if the lost one was like that. >> >> Another engineer that looked at the Rados objects last night did notice two things: >> >> - The head object had a size of 0. >> - There was an object with a ’shadow’ in its name, belonging to that path. >> >> I’m not knowledgable about Rados, so I’m not sure this is helpful. >> >>> On 18 Nov 2020, at 10:01, Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx> wrote: >>> >>> Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME (forgot the "object" there) >>> >>> On 18/11/2020 09:58, Janek Bevendorff wrote: >>>>> The object, a Docker layer, that went missing has not been touched in 2 months. It worked for a while, but then suddenly went missing. >>>> Was the object a multipart object? You can check by running radosgw-admin stat --bucket=BUCKETNAME --object=OBJECTNAME. It should say something "ns": "multipart" in the output. If it says "ns": "shadow", it's a single-part object. >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@xxxxxxx >>> To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |