Looks like I’ve now got a consistent repro scenario, please find the gory details here http://tracker.ceph.com/issues/20380 Thanks! On 20/06/17, 2:04 PM, "Pavan Rallabhandi" <PRallabhandi@xxxxxxxxxxxxxxx> wrote: Hi Orit, No, we do not use multi-site. Thanks, -Pavan. From: Orit Wasserman <owasserm@xxxxxxxxxx> Date: Tuesday, 20 June 2017 at 12:49 PM To: Pavan Rallabhandi <PRallabhandi@xxxxxxxxxxxxxxx> Cc: "ceph-users@xxxxxxxxxxxxxx" <ceph-users@xxxxxxxxxxxxxx> Subject: EXT: Re: FW: radosgw: stale/leaked bucket index entries Hi Pavan, On Tue, Jun 20, 2017 at 8:29 AM, Pavan Rallabhandi <PRallabhandi@xxxxxxxxxxxxxxx> wrote: Trying one more time with ceph-users On 19/06/17, 11:07 PM, "Pavan Rallabhandi" <PRallabhandi@xxxxxxxxxxxxxxx> wrote: On many of our clusters running Jewel (10.2.5+), am running into a strange problem of having stale bucket index entries left over for (some of the) objects deleted. Though it is not reproducible at will, it has been pretty consistent of late and am clueless at this point for the possible reasons to happen so. The symptoms are that the actual delete operation of an object is reported successful in the RGW logs, but a bucket list on the container would still show the deleted object. An attempt to download/stat of the object appropriately results in a failure. No failures are seen in the respective OSDs where the bucket index object is located. And rebuilding the bucket index by running ‘radosgw-admin bucket check –fix’ would fix the issue. Though I could simulate the problem by instrumenting the code, to not to have invoked `complete_del` on the bucket index op https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rados.cc#L8793, but that call is always seem to be made unless there is a cascading error from the actual delete operation of the object, which doesn’t seem to be the case here. I wanted to know the possible reasons where the bucket index would be left in such limbo, any pointers would be much appreciated. FWIW, we are not sharding the buckets and very recently I’ve seen this happen with buckets having as low as < 10 objects, and we are using swift for all the operations. Do you use multisite? Regards, Orit Thanks, -Pavan. _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |