Odd, it looks like it's stuck in the "MOVING" state. Perhaps the "rbd trash mv" command was aborted mid-operation? The way to work around this issue is as follows: $ rados -p volumes getomapval rbd_trash id_5afa5e5a07b8bc key_file $ hexedit key_file ## CHANGE LAST BYTE FROM '01' to '00' $ rados -p volumes setomapval rbd_trash id_5afa5e5a07b8bc --input-file key_file $ rbd trash rm --pool volumes 5afa5e5a07b8bc I'll open a ticket to expose the ability to run "rbd trash mv --image-id <XYZ>" to workaround an interrupted move operation in the future and to cleanup the error text. On Wed, Mar 10, 2021 at 8:37 AM Enrico Bocchi <enrico.bocchi@xxxxxxx> wrote: > > Hello Jason, > > # rados -p volumes listomapvals rbd_trash > id_5afa5e5a07b8bc > value (71 bytes) : > 00000000 02 01 41 00 00 00 00 2b 00 00 00 76 6f 6c 75 6d > |..A....+...volum| > 00000010 65 2d 30 32 64 39 35 39 66 65 2d 61 36 39 33 2d > |e-02d959fe-a693-| > 00000020 34 61 63 62 2d 39 35 65 32 2d 63 61 30 34 62 39 > |4acb-95e2-ca04b9| > 00000030 36 35 33 38 39 62 12 05 2a 60 09 c5 d4 16 12 05 > |65389b..*`......| > 00000040 2a 60 09 c5 d4 16 01 |*`.....| > 00000047 > > Hope that helps. > Cheers, > Enrico > > > On 10/03/2021 14:31, Jason Dillaman wrote: > > Can you provide the output from "rados -p volumes listomapvals rbd_trash"? > > > > On Wed, Mar 10, 2021 at 8:03 AM Enrico Bocchi <enrico.bocchi@xxxxxxx> wrote: > >> Hello everyone, > >> > >> We have an unpurgeable image living in the trash of one of our clusters: > >> # rbd --pool volumes trash ls > >> 5afa5e5a07b8bc volume-02d959fe-a693-4acb-95e2-ca04b965389b > >> > >> If we try to purge the whole trash it says the image is being restored > >> but we have never tried to do that: > >> # rbd --pool volumes trash purge > >> Removing images: 0% complete...failed. > >> 2021-03-10 13:58:42.849 7f78b3fc9c80 -1 librbd::api::Trash: remove: > >> error: image is pending restoration. > >> > >> When trying to delete manually, it says there are some watchers, but > >> this is actually not the case: > >> > >> # rbd --pool volumes trash remove 5afa5e5a07b8bc > >> rbd: error: image still has watchers2021-03-10 14:00:21.262 7f93ee8f8c80 > >> -1 librbd::api::Trash: remove: error: image is pending restoration. > >> This means the image is still open or the client using it crashed. Try > >> again after closing/unmapping it or waiting 30s for the crashed client > >> to timeout. > >> Removing image: > >> 0% complete...failed. > >> > >> # rados listwatchers -p volumes rbd_header.5afa5e5a07b8bc > >> # > >> > >> We have tried to stat the first 10 rbd_data objects and they were all > >> deleted. > >> We know we can manually delete the omapkey from rbd_trash but we though > >> it would be better to understand how an image might get in this state. > >> Has anyone seen this before? > >> > >> Many thanks! > >> Cheers, > >> Enrico > >> > >> > >> -- > >> Enrico Bocchi > >> CERN European Laboratory for Particle Physics > >> IT - Storage Group - General Storage Services > >> Mailbox: G20500 - Office: 31-2-010 > >> 1211 Genève 23 > >> Switzerland > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users@xxxxxxx > >> To unsubscribe send an email to ceph-users-leave@xxxxxxx > > > > > -- > Enrico Bocchi > CERN European Laboratory for Particle Physics > IT - Storage Group - General Storage Services > Mailbox: G20500 - Office: 31-2-010 > 1211 Genève 23 > Switzerland > -- Jason _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |