Object index database get corrupted and no ones can fix. We wipped a 500TB cluster years ago and move out ceph due this orphans bugs. After move all our data we saw in disk more than 100TB data unable to be deleted by ceph, also know as orphans... no sense. We expended thousand hours with this bug, the best solution replicate valid data to a new ceph cluster. Some providers solve this with x4 replica but no money sense. Regards, Manuel CONFIDENTIALITY NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged, proprietary and/or confidential information intended solely for the use of the addressee. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution, duplication or other use of this message and/or its attachments is strictly prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message and its attachments. Thank you. No imprimas si no es necesario. Protejamos el Medio Ambiente. -----Original Message----- From: Andrei Mikhailovsky <andrei@xxxxxxxxxx> Sent: martes, 3 de enero de 2023 13:46 To: ceph-users <ceph-users@xxxxxxx> Subject: rgw - unable to remove some orphans Happy New Year everyone! I have a bit of an issue with removing some of the orphan objects that were generated with the rgw-orphan-list tool. Over the years rgw generated over 14 million orphans with an overall waste of over 100TB in size, considering the overall data stored in rgw was well under 10TB at max. Anyways, I have managed to remove around 12m objects over the holiday season, but there are just over 2m orphans which were not removed. Here is an example of one of the objects taken from the orphans list file: $ rados -p .rgw.buckets rm 'default.775634629.1__multipart_SQL Backups/ALL-POND-LIVE_backup_2021_05_26_204508_8473183.d20210526-u200953.bak.s26895803904.zip.0e6LO9b4w9H3HepY-3IW_JSOaysLdFs.1_92' error removing .rgw.buckets>default.775634629.1__shadow_SQL Backups/ALL-POND-LIVE_backup_2021_05_26_204508_8473183.d20210526-u200953.bak.s26895803904.zip.0e6LO9b4w9H3HepY-3IW_JSOaysLdFs.1_92: (2) No such file or directory Checking the presence of the object with the rados tool shows that the object is there. $ cat orphan-list-20230103105849.out |grep -a JSOaysLdFs |grep -a 92 default.775634629.1__shadow_SQL Backups/ALL-POND-LIVE_backup_2021_05_26_204508_8473183.d20210526-u200953.bak.s26895803904.zip.0e6LO9b4w9H3HepY-3IW_JSOaysLdFs.1_92 $ cat rados-20230103105849.intermediate |grep -a JSOaysLdFs |grep -a 92 default.775634629.1__shadow_SQL Backups/ALL-POND-LIVE_backup_2021_05_26_204508_8473183.d20210526-u200953.bak.s26895803904.zip.0e6LO9b4w9H3HepY-3IW_JSOaysLdFs.1_92 Why can't I remove it? I have around 2m objects which can't be removed. What can I do to remove them? Thanks Andrei _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |