Re: NoSuchKey on key that is visible in s3 list/radosgw bk

[Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx>]

> 1. It seems like those reporting this issue are seeing it strictly 
> after upgrading to Octopus. From what version did each of these sites 
> upgrade to Octopus? From Nautilus? Mimic? Luminous?

I upgraded from the latest Luminous release.


> 2. Does anyone have any lifecycle rules on a bucket experiencing this 
> issue? If so, please describe.

Nope.


> 3. Is anyone making copies of the affected objects (to same or to a 
> different bucket) prior to seeing the issue? And if they are making 
> copies, does the destination bucket have lifecycle rules? And if they 
> are making copies, are those copies ever being removed?

We are not making copies, but we have bucket ACLs in place, which allow 
different users to access the objects. I doubt this is the problem 
though, otherwise we probably would have lost terabytes upon terabytes 
and not 16 objects so far.

> 4. Is anyone experiencing this issue willing to run their RGWs with 
> 'debug_ms=1'? That would allow us to see a request from an RGW to 
> either remove a tail object or decrement its reference counter (and 
> when its counter reaches 0 it will be deleted).

I haven't had any new data loss in the last few days (at least I think 
so, I read 1byte from all objects, but didn't compare checksums, so I 
cannot say if all objects are complete, but at least all are there).


> Thanks,
>
> Eric
>
>
> > On Nov 12, 2020, at 4:54 PM, huxiaoyu@xxxxxxxxxxxx 
> > <mailto:huxiaoyu@xxxxxxxxxxxx> wrote:
> >
> > Looks like this is a very dangerous bug for data safety. Hope the bug 
> > would be quickly identified and fixed.
> >
> > best regards,
> >
> > Samuel
> >
> >
> >
> > huxiaoyu@xxxxxxxxxxxx <mailto:huxiaoyu@xxxxxxxxxxxx>
> >
> > From: Janek Bevendorff
> > Date: 2020-11-12 18:17
> > To:huxiaoyu@xxxxxxxxxxxx <mailto:huxiaoyu@xxxxxxxxxxxx>; EDH - Manuel 
> > Rios; Rafael Lopez
> > CC: Robin H. Johnson; ceph-users
> > Subject: Re:  Re: NoSuchKey on key that is visible in s3 
> > list/radosgw bk
> > I have never seen this on Luminous. I recently upgraded to Octopus 
> > and the issue started occurring only few weeks later.
> >
> > On 12/11/2020 16:37, huxiaoyu@xxxxxxxxxxxx 
> > <mailto:huxiaoyu@xxxxxxxxxxxx> wrote:
> > which Ceph versions are affected by this RGW bug/issues? Luminous, 
> > Mimic, Octupos, or the latest?
> >
> > any idea?
> >
> > samuel
> >
> >
> >
> > huxiaoyu@xxxxxxxxxxxx <mailto:huxiaoyu@xxxxxxxxxxxx>
> >
> > From: EDH - Manuel Rios
> > Date: 2020-11-12 14:27
> > To: Janek Bevendorff; Rafael Lopez
> > CC: Robin H. Johnson; ceph-users
> > Subject:  Re: NoSuchKey on key that is visible in s3 
> > list/radosgw bk
> > This same error caused us to wipe a full cluster of 300TB... will be 
> > related to some rados index/database bug not to s3.
> >
> > As Janek exposed is a mayor issue, because the error silent happend 
> > and you can only detect it with S3, when you're going to delete/purge 
> > a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic ..
> >
> > Hope this time dev's can take enought time to find and resolve the 
> > issue. Error happens with low ec profiles, even with replica x3 in 
> > some cases.
> >
> > Regards
> >
> >
> >
> > -----Mensaje original-----
> > De: Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx 
> > <mailto:janek.bevendorff@xxxxxxxxxxxxx>>
> > Enviado el: jueves, 12 de noviembre de 2020 14:06
> > Para: Rafael Lopez <rafael.lopez@xxxxxxxxxx 
> > <mailto:rafael.lopez@xxxxxxxxxx>>
> > CC: Robin H. Johnson <robbat2@xxxxxxxxxx 
> > <mailto:robbat2@xxxxxxxxxx>>; ceph-users <ceph-users@xxxxxxx 
> > <mailto:ceph-users@xxxxxxx>>
> > Asunto:  Re: NoSuchKey on key that is visible in s3 
> > list/radosgw bk
> >
> > Here is a bug report concerning (probably) this exact issue:
> > https://tracker.ceph.com/issues/47866
> >
> > I left a comment describing the situation and my (limited) 
> > experiences with it.
> >
> >
> > On 11/11/2020 10:04, Janek Bevendorff wrote:
> > > Yeah, that seems to be it. There are 239 objects prefixed
> > > .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none
> > > of the multiparts from the other file to be found and the head object
> > > is 0 bytes.
> > >
> > > I checked another multipart object with an end pointer of 11.
> > > Surprisingly, it had way more than 11 parts (39 to be precise) named
> > > .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I
> > > could find them in the dump at least.
> > >
> > > I have no idea why the objects disappeared. I ran a Spark job over all
> > > buckets, read 1 byte of every object and recorded errors. Of the 78
> > > buckets, two are missing objects. One bucket is missing one object,
> > > the other 15. So, luckily, the incidence is still quite low, but the
> > > problem seems to be expanding slowly.
> > >
> > >
> > > On 10/11/2020 23:46, Rafael Lopez wrote:
> > > > Hi Janek,
> > > >
> > > > What you said sounds right - an S3 single part obj won't have an S3
> > > > multipart string as part of the prefix. S3 multipart string looks
> > > > like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme".
> > > >
> > > > From memory, single part S3 objects that don't fit in a single rados
> > > > object are assigned a random prefix that has nothing to do with
> > > > the object name, and the rados tail/data objects (not the head
> > > > object) have that prefix.
> > > > As per your working example, the prefix for that would be
> > > > '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow"
> > > > objects with names containing that prefix, and if you add up the
> > > > sizes it should be the size of your S3 object.
> > > >
> > > > You should look at working and non working examples of both single
> > > > and multipart S3 objects, as they are probably all a bit different
> > > > when you look in rados.
> > > >
> > > > I agree it is a serious issue, because once objects are no longer in
> > > > rados, they cannot be recovered. If it was a case that there was a
> > > > link broken or rados objects renamed, then we could work to
> > > > recover...but as far as I can tell, it looks like stuff is just
> > > > vanishing from rados. The only explanation I can think of is some
> > > > (rgw or rados) background process is incorrectly doing something with
> > > > these objects (eg. renaming/deleting). I had thought perhaps it was a
> > > > bug with the rgw garbage collector..but that is pure speculation.
> > > >
> > > > Once you can articulate the problem, I'd recommend logging a bug
> > > > tracker upstream.
> > > >
> > > >
> > > > On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff
> > > > <janek.bevendorff@xxxxxxxxxxxxx <mailto:janek.bevendorff@xxxxxxxxxxxxx>
> > > > <mailto:janek.bevendorff@xxxxxxxxxxxxx>> wrote:
> > > >
> > > >    Here's something else I noticed: when I stat objects that work
> > > >    via radosgw-admin, the stat info contains a "begin_iter" JSON
> > > >    object with RADOS key info like this
> > > >
> > > >
> > > >                        "key": {
> > > >                            "name":
> > > >    "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz",
> > > >                            "instance": "",
> > > >                            "ns": ""
> > > >                        }
> > > >
> > > >
> > > >    and then "end_iter" with key info like this:
> > > >
> > > >
> > > >                        "key": {
> > > >                            "name":
> > > >    ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239",
> > > >                            "instance": "",
> > > >                            "ns": "shadow"
> > > >                        }
> > > >
> > > >    However, when I check the broken 0-byte object, the "begin_iter"
> > > >    and "end_iter" keys look like this:
> > > >
> > > >
> > > >                        "key": {
> > > >                            "name":
> > > >    "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1",
> > > >                            "instance": "",
> > > >                            "ns": "multipart"
> > > >                        }
> > > >
> > > >    [...]
> > > >
> > > >
> > > >                        "key": {
> > > >                            "name":
> > > >    "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19",
> > > >                            "instance": "",
> > > >                            "ns": "multipart"
> > > >                        }
> > > >
> > > >    So, it's the full name plus a suffix and the namespace is
> > > >    multipart, not shadow (or empty). This in itself may just be an
> > > >    artefact of whether the object was uploaded in one go or as a
> > > >    multipart object, but the second difference is that I cannot find
> > > >    any of the multipart objects in my pool's object name dump. I
> > > >    can, however, find the shadow RADOS object of the intact S3 object.
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Rafael Lopez*
> > > > Devops Systems Engineer
> > > > Monash University eResearch Centre
> > > >
> > > > T: +61 3 9905 9118 <tel:%2B61%203%209905%209118>
> > > > E: rafael.lopez@xxxxxxxxxx <mailto:rafael.lopez@xxxxxxxxxx> 
> > > > <mailto:rafael.lopez@xxxxxxxxxx>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx 
> > <mailto:ceph-users-leave@xxxxxxx>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx 
> > <mailto:ceph-users-leave@xxxxxxx>
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@xxxxxxx <mailto:ceph-users@xxxxxxx>
> > To unsubscribe send an email to ceph-users-leave@xxxxxxx 
> > <mailto:ceph-users-leave@xxxxxxx>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx