I have never seen this on Luminous. I recently upgraded to Octopus and
the issue started occurring only few weeks later.
On 12/11/2020 16:37, huxiaoyu@xxxxxxxxxxxx wrote:
which Ceph versions are affected by this RGW bug/issues? Luminous,
Mimic, Octupos, or the latest?
any idea?
samuel
------------------------------------------------------------------------
huxiaoyu@xxxxxxxxxxxx
*From:* EDH - Manuel Rios <mailto:mriosfer@xxxxxxxxxxxxxxxx>
*Date:* 2020-11-12 14:27
*To:* Janek Bevendorff <mailto:janek.bevendorff@xxxxxxxxxxxxx>;
Rafael Lopez <mailto:rafael.lopez@xxxxxxxxxx>
*CC:* Robin H. Johnson <mailto:robbat2@xxxxxxxxxx>; ceph-users
<mailto:ceph-users@xxxxxxx>
*Subject:* Re: NoSuchKey on key that is visible in s3
list/radosgw bk
This same error caused us to wipe a full cluster of 300TB... will
be related to some rados index/database bug not to s3.
As Janek exposed is a mayor issue, because the error silent
happend and you can only detect it with S3, when you're going to
delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related
to S3 logic ..
Hope this time dev's can take enought time to find and resolve the
issue. Error happens with low ec profiles, even with replica x3 in
some cases.
Regards
-----Mensaje original-----
De: Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx>
Enviado el: jueves, 12 de noviembre de 2020 14:06
Para: Rafael Lopez <rafael.lopez@xxxxxxxxxx>
CC: Robin H. Johnson <robbat2@xxxxxxxxxx>; ceph-users
<ceph-users@xxxxxxx>
Asunto: Re: NoSuchKey on key that is visible in s3
list/radosgw bk
Here is a bug report concerning (probably) this exact issue:
https://tracker.ceph.com/issues/47866
I left a comment describing the situation and my (limited)
experiences with it.
On 11/11/2020 10:04, Janek Bevendorff wrote:
>
> Yeah, that seems to be it. There are 239 objects prefixed
> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are
none
> of the multiparts from the other file to be found and the head
object
> is 0 bytes.
>
> I checked another multipart object with an end pointer of 11.
> Surprisingly, it had way more than 11 parts (39 to be precise)
named
> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I
> could find them in the dump at least.
>
> I have no idea why the objects disappeared. I ran a Spark job
over all
> buckets, read 1 byte of every object and recorded errors. Of the 78
> buckets, two are missing objects. One bucket is missing one object,
> the other 15. So, luckily, the incidence is still quite low, but
the
> problem seems to be expanding slowly.
>
>
> On 10/11/2020 23:46, Rafael Lopez wrote:
>> Hi Janek,
>>
>> What you said sounds right - an S3 single part obj won't have
an S3
>> multipart string as part of the prefix. S3 multipart string looks
>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme".
>>
>> From memory, single part S3 objects that don't fit in a single
rados
>> object are assigned a random prefix that has nothing to do with
>> the object name, and the rados tail/data objects (not the head
>> object) have that prefix.
>> As per your working example, the prefix for that would be
>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239)
"shadow"
>> objects with names containing that prefix, and if you add up the
>> sizes it should be the size of your S3 object.
>>
>> You should look at working and non working examples of both single
>> and multipart S3 objects, as they are probably all a bit different
>> when you look in rados.
>>
>> I agree it is a serious issue, because once objects are no
longer in
>> rados, they cannot be recovered. If it was a case that there was a
>> link broken or rados objects renamed, then we could work to
>> recover...but as far as I can tell, it looks like stuff is just
>> vanishing from rados. The only explanation I can think of is some
>> (rgw or rados) background process is incorrectly doing
something with
>> these objects (eg. renaming/deleting). I had thought perhaps it
was a
>> bug with the rgw garbage collector..but that is pure speculation.
>>
>> Once you can articulate the problem, I'd recommend logging a bug
>> tracker upstream.
>>
>>
>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff
>> <janek.bevendorff@xxxxxxxxxxxxx
>> <mailto:janek.bevendorff@xxxxxxxxxxxxx>> wrote:
>>
>> Here's something else I noticed: when I stat objects that work
>> via radosgw-admin, the stat info contains a "begin_iter" JSON
>> object with RADOS key info like this
>>
>>
>> "key": {
>> "name":
>>
"29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz",
>> "instance": "",
>> "ns": ""
>> }
>>
>>
>> and then "end_iter" with key info like this:
>>
>>
>> "key": {
>> "name":
>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239",
>> "instance": "",
>> "ns": "shadow"
>> }
>>
>> However, when I check the broken 0-byte object, the
"begin_iter"
>> and "end_iter" keys look like this:
>>
>>
>> "key": {
>> "name":
>>
"29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1",
>> "instance": "",
>> "ns": "multipart"
>> }
>>
>> [...]
>>
>>
>> "key": {
>> "name":
>>
"29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19",
>> "instance": "",
>> "ns": "multipart"
>> }
>>
>> So, it's the full name plus a suffix and the namespace is
>> multipart, not shadow (or empty). This in itself may just be an
>> artefact of whether the object was uploaded in one go or as a
>> multipart object, but the second difference is that I
cannot find
>> any of the multipart objects in my pool's object name dump. I
>> can, however, find the shadow RADOS object of the intact
S3 object.
>>
>>
>>
>>
>> --
>> *Rafael Lopez*
>> Devops Systems Engineer
>> Monash University eResearch Centre
>>
>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118>
>> E: rafael.lopez@xxxxxxxxxx <mailto:rafael.lopez@xxxxxxxxxx>
>>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
