Yeah, that seems to be it. There are 239 objects prefixed
.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none of
the multiparts from the other file to be found and the head object is 0
bytes.
I checked another multipart object with an end pointer of 11.
Surprisingly, it had way more than 11 parts (39 to be precise) named .1,
.1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I could
find them in the dump at least.
I have no idea why the objects disappeared. I ran a Spark job over all
buckets, read 1 byte of every object and recorded errors. Of the 78
buckets, two are missing objects. One bucket is missing one object, the
other 15. So, luckily, the incidence is still quite low, but the problem
seems to be expanding slowly.
On 10/11/2020 23:46, Rafael Lopez wrote:
Hi Janek,
What you said sounds right - an S3 single part obj won't have an S3
multipart string as part of the prefix. S3 multipart string looks like
"2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme".
From memory, single part S3 objects that don't fit in a single rados
object are assigned a random prefix that has nothing to do with
the object name, and the rados tail/data objects (not the head object)
have that prefix.
As per your working example, the prefix for that would be
'.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow"
objects with names containing that prefix, and if you add up the sizes
it should be the size of your S3 object.
You should look at working and non working examples of both single and
multipart S3 objects, as they are probably all a bit different when
you look in rados.
I agree it is a serious issue, because once objects are no longer in
rados, they cannot be recovered. If it was a case that there was a
link broken or rados objects renamed, then we could work to
recover...but as far as I can tell, it looks like stuff is just
vanishing from rados. The only explanation I can think of is some (rgw
or rados) background process is incorrectly doing something with these
objects (eg. renaming/deleting). I had thought perhaps it was a bug
with the rgw garbage collector..but that is pure speculation.
Once you can articulate the problem, I'd recommend logging a bug
tracker upstream.
On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff
<janek.bevendorff@xxxxxxxxxxxxx
<mailto:janek.bevendorff@xxxxxxxxxxxxx>> wrote:
Here's something else I noticed: when I stat objects that work via
radosgw-admin, the stat info contains a "begin_iter" JSON object
with RADOS key info like this
"key": {
"name":
"29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz",
"instance": "",
"ns": ""
}
and then "end_iter" with key info like this:
"key": {
"name":
".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239",
"instance": "",
"ns": "shadow"
}
However, when I check the broken 0-byte object, the "begin_iter"
and "end_iter" keys look like this:
"key": {
"name":
"29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1",
"instance": "",
"ns": "multipart"
}
[...]
"key": {
"name":
"29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19",
"instance": "",
"ns": "multipart"
}
So, it's the full name plus a suffix and the namespace is
multipart, not shadow (or empty). This in itself may just be an
artefact of whether the object was uploaded in one go or as a
multipart object, but the second difference is that I cannot find
any of the multipart objects in my pool's object name dump. I can,
however, find the shadow RADOS object of the intact S3 object.
--
*Rafael Lopez*
Devops Systems Engineer
Monash University eResearch Centre
T: +61 3 9905 9118 <tel:%2B61%203%209905%209118>
E: rafael.lopez@xxxxxxxxxx <mailto:rafael.lopez@xxxxxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
