Hello, We have a couple of RBD images in a pool that are unable to be deleted. The user attempted to delete these volumes , while we were in the middle of a ceph minor version upgrade (where ceph processes restart). I am suspecting that during one of the service restarts (probably monitor?), the image deletion only got half way, and let these images in a bad state. It looks like the images were scheduled to move to the trash (from the rbd rm), but did not make it. Their omap values still exist in the trash though. Any suggestions on how these can be cleaned up? We are running Ceph 15.2.17 (now , after the upgrade from Ceph 15.2.15/16). Thank you #rbd rm 418824e0-576a-4167-957a-5f3fa6f2693a -p <pool name> Removing image: 0% complete...failed. rbd: delete error: (117) Structure needs cleaning 2023-01-26T15:14:40.258+0000 7f2677dfb380 -1 librbd::api::Trash: remove: error: image is pending moving to the trash. #rbd -p <pool name> trash ls no output #rbd -p <pool> ls | grep -i 418824e0-576a-4167-957a-5f3fa6f2693a 418824e0-576a-4167-957a-5f3fa6f2693a #rbd info <pool name>/418824e0-576a-4167-957a-5f3fa6f2693a rbd: error opening image 418824e0-576a-4167-957a-5f3fa6f2693a: (2) No such file or directory #rados -p <pool name> listomapvals rbd_trash id_db434b7d01af76 value (64 bytes) : 00000000 02 01 3a 00 00 00 03 24 00 00 00 34 31 38 38 32 |..:....$...41882| 00000010 34 65 30 2d 35 37 36 61 2d 34 31 36 37 2d 39 35 |4e0-576a-4167-95| 00000020 37 61 2d 35 66 33 66 61 36 66 32 36 39 33 61 0e |7a-5f3fa6f2693a.| 00000030 41 22 63 06 f7 bf 23 0e 41 22 63 06 f7 bf 23 01 |A"c...#.A"c...#.| 00000040 id_db5431429f149d value (64 bytes) : 00000000 02 01 3a 00 00 00 03 24 00 00 00 33 65 31 63 65 |..:....$...3e1ce| 00000010 61 34 34 2d 30 31 63 35 2d 34 62 65 39 2d 39 31 |a44-01c5-4be9-91| 00000020 32 30 2d 37 65 64 64 62 66 37 65 39 65 32 63 0e |20-7eddbf7e9e2c.| 00000030 41 22 63 97 2b 3e 20 0e 41 22 63 97 2b 3e 20 01 |A"c.+> .A"c.+> .| 00000040 Important Notice: This email is intended to be received only by persons entitled to receive the confidential and legally privileged information it presumptively contains, and this notice constitutes identification as such. Any reading, disclosure, copying, distribution or use of this information by or to someone who is not the intended recipient, is prohibited. If you received this email in error, please notify us immediately at legal@xxxxxxxxxx, and then delete it. To opt-out of receiving emails Please click here<https://info.kaseya.com/email-subscription-center.html>. The term 'this e-mail' includes any and all attachments. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |