hi Adam, On Thu, Jun 27, 2024 at 4:41 AM Adam Prycki <aprycki@xxxxxxxxxxxxx> wrote: > > Hello, > > I have a question. Do people use rgw lifecycle policies in production? > I had big hopes for this technology bug in practice it seems to be very > unreliable. > > Recently I've been testing different pool layouts and using lifecycle > policy to move data between them. Once I've checked orphaned objects > I've discovered that my pools were full of orphaned objects. One pool > was over 1/3 orphans by volume. Orphan object belonged to data that was > moved by lifecycle. > > Yesterday I decided to recreate one of the pools with 3TiB of data. All > 3TiB was located in a single directory of some buckets. I've created a > lifecycle which should move it all to STANDARD pool and run > radosgw-admin lc process --bucket. After lifecycle finished executing > ceph pool still contained 1TiB of data. Removing objects from > rgw-orphan-list output reduced pool size to 65GiB and 17k objects. > > The 17k rados __shadow objects seem to belong to s3 objects which were > not moved by lifecycle. I tried lifecycle from radosgw-admin but > lifecycle seems to be unable to move them. s3cmd info show that they > still report old storage class. Filenames don't contain special > characters other than spaces. I have directories with sequentially named > objects, some of them cannot be moved by lifecycle. > > Deleting all the objects form original 3TiB dataset also doesn't help. > After running gc and orphan finding tool there are still 1,2k rados > objects which should have been deleted but are not considered orphans. i assume you used `radosgw-admin gc process` here - can you confirm whether you added the --include-all option? without that option, garbage collection won't delete objects newer than rgw_gc_obj_min_wait=2hours in case they're still being read. it sounds like these rados objects may still be in the gc queue, which could explain why they aren't considered orphans > > I've been testing on 18.2.2. > > Best regards > Adam Prycki > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |