hi, On Wed, 2021-11-24 at 17:16 +0530, Pritha Srivastava wrote: > Can you please open a tracker issue and attach detailed rgw logs for > objects that you know aren't getting removed from the gc queue, like > the one that you had pasted above: sure. applied for an account yesterday, but still waiting for it to be blessed by the redmine admin. ;) I will find logs for a few other objects (there are currently 30k rgw objects and 1.8M rados objects in the gc queue, I believe most of these are problematic as the deletes usually keep the gc queue almost empty, save for the nonexpired ones). note that some rgw objects are really large, i.e. have a few 10k of tail objects and a gc of such an object can take up to half an hour from what I observed. do you think that matters? > 2021-11-23T14:54:00.061+0100 7f6afa7fc700 20 garbage collection: > RGWGC::process iterating over entry tag='23d143e2-d02d-4481-ba81- > e783696ec99f.93072205.26537934^@', time=2021-11- > 21T12:01:08.225897+0100, chain.objs.size()=3 > 2021-11-23T14:54:00.061+0100 7f6afa7fc700 5 garbage collection: > RGWGC::process removing default.rgw.buckets.data:23d143e2-d02d-4481- > ba81- > e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_1 > 2021-11-23T14:54:00.753+0100 7f6afa7fc700 5 garbage collection: > RGWGC::process removing default.rgw.buckets.data:23d143e2-d02d-4481- > ba81- > e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_2 > 2021-11-23T14:54:00.753+0100 7f6afa7fc700 5 garbage collection: > RGWGC::process removing default.rgw.buckets.data:23d143e2-d02d-4481- > ba81- > e783696ec99f.43219778.5048__shadow_.fK9K7WI3BhIiUbDXoS5UAmcpYqmShR5_3 > > If you have corresponding osd logs, please attach them as well. OSDs only have normal logging, not verbose. I can check, but could you give a hint on how to find the OSD log correlated with the deletion above? the timeframe seems one pointer, but - which OSDs? > Also please add other details like - the settings that you have used. will check for anything out of the ordinary. > Did you change these settings after upgrading from nautilus to > octopus?(Seems like you didn't). no. > And also how many days after upgrading did you start seeing this > problem? the upgrade was done on september 6th and 7th. the problems started on november 10th. it was in the middle of a capacity increase (adding two OSD hosts and removing another), which finished on november 13th. > I will take a look at it asap. thank you. best, Jaka _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |