I would recommend you get a dump with rados ls -p poolname (can be
several GB, mine is 61GB) and grep (or ack, which is faster) for the
names there to get an overview of what is there and what isn't. Looking
up the names directly can easily give you the wrong picture, because it
is kinda complicated to derive the correct RADOS name from an S3 object
name and a single typo will give you a not-found error, even if the
object is there.
On 19/11/2020 09:12, Denis Krienbühl wrote:
Thanks, we are currently scanning our object storage. It looks like we can detect the missing objects that return “No Such Key” looking at all “__multipart_” objects returned by radosgw-admin bucket radoslist, and checking if they exist using rados stat. We are currently not looking at shadow objects as our approach already yields more instances of this problem.
On 19 Nov 2020, at 09:09, Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx> wrote:
- The head object had a size of 0.
- There was an object with a ’shadow’ in its name, belonging to that path.
That is normal. What is not normal is if there are NO shadow objects.
On 18/11/2020 10:06, Denis Krienbühl wrote:
It looks like a single-part object. But we did replace that object last night from backup, so I can’t know for sure if the lost one was like that.
Another engineer that looked at the Rados objects last night did notice two things:
- The head object had a size of 0.
- There was an object with a ’shadow’ in its name, belonging to that path.
I’m not knowledgable about Rados, so I’m not sure this is helpful.
On 18 Nov 2020, at 10:01, Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx> wrote:
Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME (forgot the "object" there)
On 18/11/2020 09:58, Janek Bevendorff wrote:
The object, a Docker layer, that went missing has not been touched in 2 months. It worked for a while, but then suddenly went missing.
Was the object a multipart object? You can check by running radosgw-admin stat --bucket=BUCKETNAME --object=OBJECTNAME. It should say something "ns": "multipart" in the output. If it says "ns": "shadow", it's a single-part object.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
