Hi Felix, This is weird. The most likely explanation is that this resulted from a bug that has certainly been fixed. What you try could as a starter is a ceph-bluestore-tool fsck [1] over the primary OSD of rbd_data.26f7c5d05af621.0000000000002adf to see if it lists any inconsistencies: $ cephadm shell --name osd.$i --fsid $(ceph fsid) ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-$i/ --deep yes If it doesn't then I suppose you'll have to get rid of these objects you can't 'rados rm' with the help of ceph-objectstore-tool <remove> [2]. Regards, Frédéric. [1] https://docs.ceph.com/en/latest/man/8/ceph-bluestore-tool/ [2] https://docs.ceph.com/en/latest/man/8/ceph-objectstore-tool/#removing-an-object ----- Le 31 Jan 25, à 8:29, Felix Stolte <f.stolte@xxxxxxxxxxxxx> a écrit : > Hi Frederic, > thanks for you suggestions. I took a look at all the objects and discovered the > following: > 1. rbd_id.<image_name>, rbd_header.<image_id> and rbd_object_map.<image_id> > exist only for the 3 images listed by ‚rbd ls‘ (i created a test image > yesterday). > 2. There are about 30 different image_ids with rbd_data.<image_id> Objects which > do not have an id, header oder object_map object > After that i tried to get the stats of one of the orphaned objects: > rados -p mailbox stat rbd_data.26f7c5d05af621.0000000000002adf > error stat-ing mailbox/rbd_data.26f7c5d05af621.0000000000002adf: (2) No such > file or directory > I double checked, that object name is the one listed by 'rados ls‘. What makes > it worse is that i neither can stat, get or rm the objects while they still are > counted for disk usage. We will remove the whole pool for sure, but i really > like to get to the cause of this to prevent it from happening again. >> Am 30.01.2025 um 10:54 schrieb Frédéric Nass <frederic.nass@xxxxxxxxxxxxxxxx>: >> Hi Felix, >> Every rbd_data object belongs to an image that should have : >> - an rbd_id.<image_name> object containing the image id (that you can get with >> rbd info) >> - an rbd_header object with omap atrributes you can list with listomapvals >> To identify the image name these rbd_data object belong(ed) to you could list >> all rbd_id objects in that pool and, for each one of them, print the image id >> and the image name with the below command: >> $ for rbd_id in $(rados -p $poolname ls | grep rbd_id) ; do echo "$(echo $rbd_id >> | cut -d '.' -f2) : $(rados -p $poolname get $rbd_id - | strings)" ; done >> image2 : 2a733b30debc84 >> image1 : 28d4fc1dddd922 >> It might take some time but you'd get a clearer view of what these rbd_data >> objects refer(ed) to. Also, if you can decode the timestamps in 'rados -p rbd >> listomapvals rbd_header.<id> | strings' output, you could know when each image >> was created and accessed for the last time. >> Hope that helps. >> Regards, >> Frédéric. >> PS: If you're moving away from iSCSI and only have 2 remaining images in this >> pool, you may also wait until these images are no longer in use and then detach >> them and remove the whole pool. >> ----- Le 30 Jan 25, à 9:09, Felix Stolte <f.stolte@xxxxxxxxxxxxx> a écrit : >>> Hi Frederic, >>> there is no namespace. The pool in question has the application rbd, but is not >>> the default pool named ‚rbd' >>>> Am 29.01.2025 um 11:24 schrieb Frédéric Nass <frederic.nass@xxxxxxxxxxxxxxxx>: >>>> Hi Felix, >>>> Any RADOS namespaces in that pool? You can check using either: >>>> rbd namespace ls rbd >>>> or >>>> rados stat -p rbd rbd_namespace && rados -p rbd listomapvals rbd_namespace >>>> The rbd_data objects might be linked to namespaced images that can only be >>>> listed using the command: rbd ls --namespace <namespace> >>>> I suggest checking this because the 'rbd' pool has historically been Ceph's >>>> default RBD pool, long before iSCSI began using it (in its hardcoded >>>> implementation). >>>> Might be worth checking this before taking any actions. >>>> Regards, >>>> Frédéric. >>>> ----- Le 29 Jan 25, à 8:53, Felix Stolte f.stolte@xxxxxxxxxxxxx a écrit : >>>>> Hi Alexander, >>>>> trash is empty and rbd ls only lists two images with the prefix >>>>> rbd_data.1af561611d24cf and rbd_data.ed93e6548ca56b >>>>> rados ls gives: >>>>> rbd_data.d1b81247165450.00000000000055d2 >>>>> rbd_data.32de606b8b4567.0000000000012f2f >>>>> rbd_data.ed93e6548ca56b.00000000000eef03 >>>>> rbd_data.26f7c5d05af621.0000000000002adf >>>>> …. >>>>> Am 28.01.2025 um 22:46 schrieb Alexander Patrakov <patrakov@xxxxxxxxx>: >>>>> Hi Felix, >>>>> A dumb answer first: if you know the image names, have you tried "rbd >>>>> rm $pool/$imagename"? Or, is there any reason like concerns about >>>>> iSCSI control data integrity that prevents you from trying that? >>>>> Also, have you checked the rbd trash? >>>>> On Tue, Jan 28, 2025 at 5:43 PM Stolte, Felix <f.stolte@xxxxxxxxxxxxx> wrote: >>>>> Hi guys, >>>>> we have a rbd pool we used for images exported via ceph-iscsi on a 17.2.7 >>>>> cluster. The pool uses 10 times the diskspace i would suppose it should and >>>>> after investigating we noticed a lot of rbd_data Objects which images are no >>>>> longer present. I assume that the original images were deleted using the gwcli >>>>> but not all Objects have been removed properly. >>>>> What would be the best/most secure way to get rid of these orphaned objects and >>>>> reclaim the diskspace? >>>>> Best regards >>>>> Felix >>>>> --------------------------------------------------------------------------------------------- >>>>> --------------------------------------------------------------------------------------------- >>>>> Forschungszentrum Juelich GmbH >>>>> 52425 Juelich >>>>> Sitz der Gesellschaft: Juelich >>>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>>>> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >>>>> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >>>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens >>>>> --------------------------------------------------------------------------------------------- >>>>> --------------------------------------------------------------------------------------------- >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>>>> -- >>>>> Alexander Patrakov >>>>> mit freundlichem Gruß >>>>> Felix Stolte >>>>> IT-Services >>>>> --------------------------------------------------------------------------------------------- >>>>> --------------------------------------------------------------------------------------------- >>>>> Forschungszentrum Juelich GmbH >>>>> 52425 Juelich >>>>> Sitz der Gesellschaft: Juelich >>>>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>>>> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >>>>> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >>>>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens >>>>> --------------------------------------------------------------------------------------------- >>>>> --------------------------------------------------------------------------------------------- >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@xxxxxxx >>>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx >>> mit freundlichem Gruß >>> Felix Stolte >>> IT-Services >>> --------------------------------------------------------------------------------------------- >>> --------------------------------------------------------------------------------------------- >>> Forschungszentrum Juelich GmbH >>> 52425 Juelich >>> Sitz der Gesellschaft: Juelich >>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 >>> Vorsitzender des Aufsichtsrats: MinDir Stefan Müller >>> Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), >>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens >>> --------------------------------------------------------------------------------------------- >>> --------------------------------------------------------------------------------------------- > mit freundlichem Gruß > Felix Stolte > IT-Services > --------------------------------------------------------------------------------------------- > --------------------------------------------------------------------------------------------- > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Stefan Müller > Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr. Ir. Pieter Jansens > --------------------------------------------------------------------------------------------- > --------------------------------------------------------------------------------------------- _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |