Hi Casey,
I did use full `radosgw-admin gc process --include-all`
I know about that background gc delay.
Is running `radosgw-admin gc process --include-all` from terminal any
different than gc process running in the background? I wonder if I
should use it while trying to recreate this issue.
Best regards
Adam Prycki
W dniu 2024-06-27 20:58, Casey Bodley napisał(a):
hi Adam,
On Thu, Jun 27, 2024 at 4:41 AM Adam Prycki <aprycki@xxxxxxxxxxxxx>
wrote:
Hello,
I have a question. Do people use rgw lifecycle policies in production?
I had big hopes for this technology bug in practice it seems to be
very
unreliable.
Recently I've been testing different pool layouts and using lifecycle
policy to move data between them. Once I've checked orphaned objects
I've discovered that my pools were full of orphaned objects. One pool
was over 1/3 orphans by volume. Orphan object belonged to data that
was
moved by lifecycle.
Yesterday I decided to recreate one of the pools with 3TiB of data.
All
3TiB was located in a single directory of some buckets. I've created a
lifecycle which should move it all to STANDARD pool and run
radosgw-admin lc process --bucket. After lifecycle finished executing
ceph pool still contained 1TiB of data. Removing objects from
rgw-orphan-list output reduced pool size to 65GiB and 17k objects.
The 17k rados __shadow objects seem to belong to s3 objects which were
not moved by lifecycle. I tried lifecycle from radosgw-admin but
lifecycle seems to be unable to move them. s3cmd info show that they
still report old storage class. Filenames don't contain special
characters other than spaces. I have directories with sequentially
named
objects, some of them cannot be moved by lifecycle.
Deleting all the objects form original 3TiB dataset also doesn't help.
After running gc and orphan finding tool there are still 1,2k rados
objects which should have been deleted but are not considered orphans.
i assume you used `radosgw-admin gc process` here - can you confirm
whether you added the --include-all option? without that option,
garbage collection won't delete objects newer than
rgw_gc_obj_min_wait=2hours in case they're still being read. it sounds
like these rados objects may still be in the gc queue, which could
explain why they aren't considered orphans
I've been testing on 18.2.2.
Best regards
Adam Prycki
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
