I’m wondering if anyone experiencing this bug would mind running `radosgw-admin gc list --include-all` on a schedule and saving the results. I’d like to know whether these tail objects are getting removed by the gc process. If we find that that’s the case then there’s the issue of how they got on the gc list. Eric > On Nov 16, 2020, at 3:48 AM, Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx> wrote: > > As noted in the bug report, the issue has affected only multipart objects at this time. I have added some more remarks there. > > And yes, multipart objects tend to have 0 byte head objects in general. The affected objects are simply missing all shadow objects, leaving us with nothing but the empty head object and a few metadata. > > > > On 13/11/2020 20:14, Eric Ivancich wrote: >> Thank you for the answers to those questions, Janek. >> >> And in case anyone hasn’t seen it, we do have a tracker for this issue: >> >> https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> >> >> We may want to move most of the conversation to the comments there, so everything’s together. >> >> I do want to follow up on your answer to Question 4, Janek: >> >>> On Nov 13, 2020, at 12:22 PM, Janek Bevendorff <janek.bevendorff@xxxxxxxxxxxxx <mailto:janek.bevendorff@xxxxxxxxxxxxx>> wrote: >>>> >>>> 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). >>> I haven't had any new data loss in the last few days (at least I think so, I read 1byte from all objects, but didn't compare checksums, so I cannot say if all objects are complete, but at least all are there). >>> >> With multipart uploads I believe this is a sufficient test, as the first bit of data is in the first tail object, and it’s tail objects that seem to be disappearing. >> >> However if the object is not uploaded via multipart and if it does have tail (_shadow_) objects, then the initial data is stored in the head object. So this test would not be truly diagnostic. This could be done with a large object, for example, with `s3cmd put --disable-multipart …`. >> >> Eric >> >> -- >> J. Eric Ivancich >> he / him / his >> Red Hat Storage >> Ann Arbor, Michigan, USA _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |