Hi Janek,
What you said sounds right - an S3 single part obj won't have an S3
multipart string as part of the prefix. S3 multipart string looks like
"2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme".
>From memory, single part S3 objects that don't fit in a single rados object
are assigned a random prefix that has nothing to do with the object name,
and the rados tail/data objects (not the head object) have that prefix.
As per your working example, the prefix for that would be
'.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow"
objects with names containing that prefix, and if you add up the sizes it
should be the size of your S3 object.
You should look at working and non working examples of both single and
multipart S3 objects, as they are probably all a bit different when you
look in rados.
I agree it is a serious issue, because once objects are no longer in rados,
they cannot be recovered. If it was a case that there was a link broken or
rados objects renamed, then we could work to recover...but as far as I can
tell, it looks like stuff is just vanishing from rados. The only
explanation I can think of is some (rgw or rados) background process is
incorrectly doing something with these objects (eg. renaming/deleting). I
had thought perhaps it was a bug with the rgw garbage collector..but that
is pure speculation.
Once you can articulate the problem, I'd recommend logging a bug tracker
upstream.
On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff <
janek.bevendorff@xxxxxxxxxxxxx> wrote:
> Here's something else I noticed: when I stat objects that work via
> radosgw-admin, the stat info contains a "begin_iter" JSON object with
> RADOS key info like this
>
>
> "key": {
> "name":
> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz",
> "instance": "",
> "ns": ""
> }
>
>
> and then "end_iter" with key info like this:
>
>
> "key": {
> "name": ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239",
> "instance": "",
> "ns": "shadow"
> }
>
> However, when I check the broken 0-byte object, the "begin_iter" and
> "end_iter" keys look like this:
>
>
> "key": {
> "name":
> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1",
> "instance": "",
> "ns": "multipart"
> }
>
> [...]
>
>
> "key": {
> "name":
> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19",
> "instance": "",
> "ns": "multipart"
> }
>
> So, it's the full name plus a suffix and the namespace is multipart, not
> shadow (or empty). This in itself may just be an artefact of whether the
> object was uploaded in one go or as a multipart object, but the second
> difference is that I cannot find any of the multipart objects in my pool's
> object name dump. I can, however, find the shadow RADOS object of the
> intact S3 object.
>
>
--
*Rafael Lopez*
Devops Systems Engineer
Monash University eResearch Centre
T: +61 3 9905 9118
E: rafael.lopez@xxxxxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
