Thanks Frédéric, we’ve done that in the meantime to work around issue #47866. The error has been reproduced and there’s a PR associated with the issue: https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> Cheers, Denis > On 23 Nov 2020, at 11:56, Frédéric Nass <frederic.nass@xxxxxxxxxxxxxxxx> wrote: > > Hi Denis, > > You might want to look at rgw_gc_obj_min_wait from [1] and try increasing the default value of 7200s (2 hours) to whatever suits your need < 2^64. > Just remind that at some point you'll have to get these objects processed by the gc. Or manually through the API [2]. > > One thing that comes to mind regarding the "last night's missing object" is maybe it was multi-part re-written and the re-write failed somehow and the object was then enlisted by the gc. But that supposes this particular object sometimes gets re-written which may not be the case. > > Regards, > > Frédéric. > > [1] https://docs.ceph.com/en/latest/radosgw/config-ref/ > [2] https://docs.ceph.com/en/latest/dev/radosgw/admin/adminops_nonimplemented/#manually-processes-garbage-collection-items > > Le 18/11/2020 à 11:27, Denis Krienbühl a écrit : >> By the way, since there’s some probability that this is a GC refcount issue, would it be possible and sane to somehow slow the GC down or disable it altogether? Is that something we could implement on our end as a stop-gap measure to prevent dataloss? >> >>> On 18 Nov 2020, at 10:46, Denis Krienbühl <denis@xxxxxxx> wrote: >>> >>> I can now confirm that last night’s missing object was a multi-part file. >>> >>>> On 18 Nov 2020, at 10:01, Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx> wrote: >>>> >>>> Sorry, it's radosgw-admin object stat --bucket=BUCKETNAME --object=OBJECTNAME (forgot the "object" there) >>>> >>>> On 18/11/2020 09:58, Janek Bevendorff wrote: >>>>>> The object, a Docker layer, that went missing has not been touched in 2 months. It worked for a while, but then suddenly went missing. >>>>> Was the object a multipart object? You can check by running radosgw-admin stat --bucket=BUCKETNAME --object=OBJECTNAME. It should say something "ns": "multipart" in the output. If it says "ns": "shadow", it's a single-part object. >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >> _______________________________________________ >> ceph-users mailing list -- ceph-users@xxxxxxx >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |