Here is a bug report concerning (probably) this exact issue:
https://tracker.ceph.com/issues/47866
I left a comment describing the situation and my (limited) experiences
with it.
On 11/11/2020 10:04, Janek Bevendorff wrote:
Yeah, that seems to be it. There are 239 objects prefixed
.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none
of the multiparts from the other file to be found and the head object
is 0 bytes.
I checked another multipart object with an end pointer of 11.
Surprisingly, it had way more than 11 parts (39 to be precise) named
.1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I
could find them in the dump at least.
I have no idea why the objects disappeared. I ran a Spark job over all
buckets, read 1 byte of every object and recorded errors. Of the 78
buckets, two are missing objects. One bucket is missing one object,
the other 15. So, luckily, the incidence is still quite low, but the
problem seems to be expanding slowly.
On 10/11/2020 23:46, Rafael Lopez wrote:
Hi Janek,
What you said sounds right - an S3 single part obj won't have an S3
multipart string as part of the prefix. S3 multipart string looks
like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme".
From memory, single part S3 objects that don't fit in a single rados
object are assigned a random prefix that has nothing to do with
the object name, and the rados tail/data objects (not the head
object) have that prefix.
As per your working example, the prefix for that would be
'.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow"
objects with names containing that prefix, and if you add up the
sizes it should be the size of your S3 object.
You should look at working and non working examples of both single
and multipart S3 objects, as they are probably all a bit different
when you look in rados.
I agree it is a serious issue, because once objects are no longer in
rados, they cannot be recovered. If it was a case that there was a
link broken or rados objects renamed, then we could work to
recover...but as far as I can tell, it looks like stuff is just
vanishing from rados. The only explanation I can think of is some
(rgw or rados) background process is incorrectly doing something with
these objects (eg. renaming/deleting). I had thought perhaps it was a
bug with the rgw garbage collector..but that is pure speculation.
Once you can articulate the problem, I'd recommend logging a bug
tracker upstream.
On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff
<janek.bevendorff@xxxxxxxxxxxxx
<mailto:janek.bevendorff@xxxxxxxxxxxxx>> wrote:
Here's something else I noticed: when I stat objects that work
via radosgw-admin, the stat info contains a "begin_iter" JSON
object with RADOS key info like this
"key": {
"name":
"29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz",
"instance": "",
"ns": ""
}
and then "end_iter" with key info like this:
"key": {
"name":
".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239",
"instance": "",
"ns": "shadow"
}
However, when I check the broken 0-byte object, the "begin_iter"
and "end_iter" keys look like this:
"key": {
"name":
"29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1",
"instance": "",
"ns": "multipart"
}
[...]
"key": {
"name":
"29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19",
"instance": "",
"ns": "multipart"
}
So, it's the full name plus a suffix and the namespace is
multipart, not shadow (or empty). This in itself may just be an
artefact of whether the object was uploaded in one go or as a
multipart object, but the second difference is that I cannot find
any of the multipart objects in my pool's object name dump. I
can, however, find the shadow RADOS object of the intact S3 object.
--
*Rafael Lopez*
Devops Systems Engineer
Monash University eResearch Centre
T: +61 3 9905 9118 <tel:%2B61%203%209905%209118>
E: rafael.lopez@xxxxxxxxxx <mailto:rafael.lopez@xxxxxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
