I don’t believe there is any tooling to find and clean orphaned bucket index shards. So if you’re certain they’re no longer needed, you can use `rados` commands to remove the objects. Eric (he/him) > On Sep 27, 2022, at 2:37 AM, Yuji Ito (伊藤 祐司) <yuji-ito@xxxxxxxxxxxx> wrote: > > Hi, > > I have encountered a problem after deleting an RGW bucket. There seem to be remaining bucket index shard objects. Could you tell me the desired way to delete these objects? Is it OK to just delete these objects? Or should I use some dedicated ceph commands? I couldn't found how to do it in the official document. > > Environment: > Rook: 1.9.6 > Ceph: 16.2.10 > > Here is a detailed information: > I got the following HEALTH_WARN after deleting a RGW bucket. > > ``` > $ kubectl exec -n ceph-poc deploy/rook-ceph-tools -- ceph health detail > HEALTH_WARN 35 large omap objects > [WRN] LARGE_OMAP_OBJECTS: 35 large omap objects > 35 large objects found in pool 'ceph-poc-object-store-ssd-index.rgw.buckets.index' > Search the cluster log for 'Large omap object found' for more details. > ``` > > I tried `bilog trim` and `stale-instance delete` commands with reffering to the following document. > - https://access.redhat.com/solutions/6450561 > - https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html-single/object_gateway_guide_for_ubuntu/index#cleaning-stale-instances-after-resharding-rgw > > Then I ran deep-scrub and this warning was disappeared. However, this warning appeared later. As a result of investigation, I found the bucket index shard objects of deleted bucket still exist. > > There were two buckets. > > ``` > $ kubectl exec -it -n ceph-poc deploy/rook-ceph-tools -- radosgw-admin bucket stats | jq '.[] | {"bucket": .bucket, "id": .id}' | jq . > { > "bucket": "csa-large-omap-9332ba5c-3cb5-4ff7-98cf-1729b44b954c", > "id": "83a2aeca-b5a0-46b2-843b-fb34884bb148.62065601.1" > } > { > "bucket": "rook-ceph-bucket-checker-dfef5d3c-036a-428a-b4df-ae6be5d5c41a", > "id": "83a2aeca-b5a0-46b2-843b-fb34884bb148.53178977.1" > } > ``` > > However, there were three sets of bucket index shard objects. > > ``` > $ kubectl exec -n ceph-poc deploy/rook-ceph-tools -- rados ls --pool ceph-poc-object-store-ssd-index.rgw.buckets.index | sort > .dir.83a2aeca-b5a0-46b2-843b-fb34884bb148.14548925.2.0 > <...snip...> > .dir.83a2aeca-b5a0-46b2-843b-fb34884bb148.14548925.2.9 > .dir.83a2aeca-b5a0-46b2-843b-fb34884bb148.53178977.1.0 > <...snip...> > .dir.83a2aeca-b5a0-46b2-843b-fb34884bb148.53178977.1.9 > .dir.83a2aeca-b5a0-46b2-843b-fb34884bb148.62065601.1.0 > <...snip...> > .dir.83a2aeca-b5a0-46b2-843b-fb34884bb148.62065601.1.9 > ``` > > I would like to delete the above unused objects by `rados rm` command. But I'm not sure whether this operation is safe or not. I would like to know how to manually delete them and the procedure to do so. > > Thanks, > Yuji > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |