Hi Felix, Can you try the below command? $ rados -p mailbox listsnaps rbd_data.26f7c5d05af621.0000000000002adf rbd_data.26f7c5d05af621.0000000000002adf: cloneid snaps size overlap 3 3 4194304 [] head - 4194304 <---- Do you see this line? If you don't see the 'head' line, then you're probably facing the orphan clones AKA leaked snapshots bug described here [1], that was fixed by [2]. To get rid of these orphan clones, you need run to run the below command on the pool that we requeue orphan objects for being snap trimmed. See [3] for details. $ ceph osd pool force-remove-snap cephfs_data_ec Note that : 1. there's a --dry-run option you can use. 2. This command should only be run ONCE. Running it twice or more is useless and can lead to OSDs crashing (restarting fine, but still... crashing at the same time). Let us know how it goes. Cheers, Frédéric. [1] https://tracker.ceph.com/issues/64646 [2] https://github.com/ceph/ceph/pull/55841 [3] https://github.com/ceph/ceph/pull/53545 ----- Le 31 Jan 25, à 9:54, Frédéric Nass frederic.nass@xxxxxxxxxxxxxxxx a écrit : > Hi Felix, > > This is weird. The most likely explanation is that this resulted from a bug that > has certainly been fixed. > > What you try could as a starter is a ceph-bluestore-tool fsck [1] over the > primary OSD of rbd_data.26f7c5d05af621.0000000000002adf to see if it lists any > inconsistencies: > > $ cephadm shell --name osd.$i --fsid $(ceph fsid) ceph-bluestore-tool fsck > --path /var/lib/ceph/osd/ceph-$i/ --deep yes > > If it doesn't then I suppose you'll have to get rid of these objects you can't > 'rados rm' with the help of ceph-objectstore-tool <remove> [2]. > > Regards, > Frédéric. > > [1] https://docs.ceph.com/en/latest/man/8/ceph-bluestore-tool/ > [2] > https://docs.ceph.com/en/latest/man/8/ceph-objectstore-tool/#removing-an-object > > ----- Le 31 Jan 25, à 8:29, Felix Stolte <f.stolte@xxxxxxxxxxxxx> a écrit : > >> Hi Frederic, >> thanks for you suggestions. I took a look at all the objects and discovered the >> following: > >> 1. rbd_id.<image_name>, rbd_header.<image_id> and rbd_object_map.<image_id> >> exist only for the 3 images listed by ‚rbd ls‘ (i created a test image >> yesterday). > >> 2. There are about 30 different image_ids with rbd_data.<image_id> Objects which >> do not have an id, header oder object_map object > >> After that i tried to get the stats of one of the orphaned objects: > >> rados -p mailbox stat rbd_data.26f7c5d05af621.0000000000002adf >> error stat-ing mailbox/rbd_data.26f7c5d05af621.0000000000002adf: (2) No such >> file or directory >> I double checked, that object name is the one listed by 'rados ls‘. What makes >> it worse is that i neither can stat, get or rm the objects while they still are >> counted for disk usage. We will remove the whole pool for sure, but i really >> like to get to the cause of this to prevent it from happening again. > >>> Am 30.01.2025 um 10:54 schrieb Frédéric Nass <frederic.nass@xxxxxxxxxxxxxxxx>: > >>> Hi Felix, > >>> Every rbd_data object belongs to an image that should have : > >>> - an rbd_id.<image_name> object containing the image id (that you can get with >>> rbd info) >>> - an rbd_header object with omap atrributes you can list with listomapvals > >>> To identify the image name these rbd_data object belong(ed) to you could list >>> all rbd_id objects in that pool and, for each one of them, print the image id >>> and the image name with the below command: > >>> $ for rbd_id in $(rados -p $poolname ls | grep rbd_id) ; do echo "$(echo $rbd_id >>> | cut -d '.' -f2) : $(rados -p $poolname get $rbd_id - | strings)" ; done >>> image2 : 2a733b30debc84 >>> image1 : 28d4fc1dddd922 > >>> It might take some time but you'd get a clearer view of what these rbd_data >>> objects refer(ed) to. Also, if you can decode the timestamps in 'rados -p rbd >>> listomapvals rbd_header.<id> | strings' output, you could know when each image >>> was created and accessed for the last time. > >>> Hope that helps. > >>> Regards, >>> Frédéric. > >>> PS: If you're moving away from iSCSI and only have 2 remaining images in this >>> pool, you may also wait until these images are no longer in use and then detach >>> them and remove the whole pool. > >>> ----- Le 30 Jan 25, à 9:09, Felix Stolte <f.stolte@xxxxxxxxxxxxx> a écrit : > >>>> Hi Frederic, >>>> there is no namespace. The pool in question has the application rbd, but is not >>>> the default pool named ‚rbd' > >>>>> Am 29.01.2025 um 11:24 schrieb Frédéric Nass <frederic.nass@xxxxxxxxxxxxxxxx>: > >>>>> Hi Felix, > >>>>> Any RADOS namespaces in that pool? You can check using either: > >>>>> rbd namespace ls rbd > >>>>> or > >>>>> rados stat -p rbd rbd_namespace && rados -p rbd listomapvals rbd_namespace > >>>>> The rbd_data objects might be linked to namespaced images that can only be >>>>> listed using the command: rbd ls --namespace <namespace> >>>>> I suggest checking this because the 'rbd' pool has historically been Ceph's >>>>> default RBD pool, long before iSCSI began using it (in its hardcoded >>>>> implementation). > >>>>> Might be worth checking this before taking any actions. > >>>>> Regards, >>>>> Frédéric. > >>>>> ----- Le 29 Jan 25, à 8:53, Felix Stolte f.stolte@xxxxxxxxxxxxx a écrit : > >>>>>> Hi Alexander, > >>>>>> trash is empty and rbd ls only lists two images with the prefix >>>>>> rbd_data.1af561611d24cf and rbd_data.ed93e6548ca56b > >>>>>> rados ls gives: > >>>>>> rbd_data.d1b81247165450.00000000000055d2 >>>>>> rbd_data.32de606b8b4567.0000000000012f2f >>>>>> rbd_data.ed93e6548ca56b.00000000000eef03 >>>>>> rbd_data.26f7c5d05af621.0000000000002adf >>>>>> …. > >>>>>> Am 28.01.2025 um 22:46 schrieb Alexander Patrakov <patrakov@xxxxxxxxx>: > >>>>>> Hi Felix, > >>>>>> A dumb answer first: if you know the image names, have you tried "rbd >>>>>> rm $pool/$imagename"? Or, is there any reason like concerns about >>>>>> iSCSI control data integrity that prevents you from trying that? > >>>>>> Also, have you checked the rbd trash? > >>>>>> On Tue, Jan 28, 2025 at 5:43 PM Stolte, Felix <f.stolte@xxxxxxxxxxxxx> wrote: > >>>>>> Hi guys, > >>>>>> we have a rbd pool we used for images exported via ceph-iscsi on a 17.2.7 >>>>>> cluster. The pool uses 10 times the diskspace i would suppose it should and >>>>>> after investigating we noticed a lot of rbd_data Objects which images are no >>>>>> longer present. I assume that the original images were deleted using the gwcli >>>>>> but not all Objects have been removed properly. > >>>>>> What would be the best/most secure way to get rid of these orphaned objects and >>>>>> reclaim the diskspace? > >>>>>> Best regards >>>>>> Felix > >>>>>> --------------------------------------------------------------------------------------------- >>>>>> --------------------------------------------------------------------------------------------- >>>>>> Forschungszentrum Juelich GmbH >>>>>> 52425 Juelich >>>>>> Sitz der Gesellschaft: Juelich >>>>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>>>>> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >>>>>> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >>>>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens >>>>>> --------------------------------------------------------------------------------------------- >>>>>> --------------------------------------------------------------------------------------------- > >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >>>>>> -- >>>>>> Alexander Patrakov > >>>>>> mit freundlichem Gruß >>>>>> Felix Stolte > >>>>>> IT-Services > >>>>>> --------------------------------------------------------------------------------------------- >>>>>> --------------------------------------------------------------------------------------------- >>>>>> Forschungszentrum Juelich GmbH >>>>>> 52425 Juelich >>>>>> Sitz der Gesellschaft: Juelich >>>>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>>>>> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >>>>>> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >>>>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens >>>>>> --------------------------------------------------------------------------------------------- >>>>>> --------------------------------------------------------------------------------------------- > >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx > >>>> mit freundlichem Gruß >>>> Felix Stolte > >>>> IT-Services > >>>> --------------------------------------------------------------------------------------------- >>>> --------------------------------------------------------------------------------------------- >>>> Forschungszentrum Juelich GmbH >>>> 52425 Juelich >>>> Sitz der Gesellschaft: Juelich >>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>>> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >>>> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens >>>> --------------------------------------------------------------------------------------------- >>>> --------------------------------------------------------------------------------------------- > >> mit freundlichem Gruß >> Felix Stolte > >> IT-Services > >> --------------------------------------------------------------------------------------------- >> --------------------------------------------------------------------------------------------- >> Forschungszentrum Juelich GmbH >> 52425 Juelich >> Sitz der Gesellschaft: Juelich >> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >> Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens >> --------------------------------------------------------------------------------------------- >> --------------------------------------------------------------------------------------------- > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |