If you read my previous email, you will see that i noted that the string IS GFID and not the name of the file :) You can find the name following the procedure at: https://docs.gluster.org/en/latest/Troubleshooting/gfid-to-path/ Of course, that will be slow for all entries in .glusterfs and you will need to create a script to match all gfids to brick path. I guess the fastest way to find the deleted files (As far as I understood they were deleted on the brick directly and entries in .glusterfs were left) is to create a script that: 0.Create a ramfs for the files: findmnt /mnt || mount -t ramfs -o size=128MB - /mnt 1. Get all inodes ionice -c 2 -n 7 nice -n 15 find /full/path/to/brick -type f -exec ls -i {} \; >/mnt/data 2. Get only the inodes: nice -n 15 awk '{print $1}' /mnt/data > /mnt/inode_only 3. Now the fun starts now-> find inodes that are not duplicate: nice -n 15 uniq -u /mnt/inode_only > /mnt/gfid-only 4. Once you have the inodes, you can verify that they do exists only in .gluster dir for i in $(cat /mnt/gfid-only); do ionice -c 2 -n 7 nice -n 15 find /path/to/.glusterfs -inum $i ; echo;echo; done 5. If it's OK -> delete for i in $(cat /mnt/gfid-only); do ionice -c 2 -n 7 nice -n 15 find /path/to/brick -inum $i -delete ; done Last , repeat on all bricks Good luck! P.S.: Consider creating a gluster snapshot before that - just in case... Better safe than sorry. P.S: If you think that you got enough resources, you can remove ionice/nice stuff . They are just to guarantee you won't eat too many resources. Best Regards, Strahil Nikolov На 8 август 2020 г. 18:02:10 GMT+03:00, Mathias Waack <mathias.waack@xxxxxxxxxxxxxxx> написа: >So b53c8e46-068b-4286-94a6-7cf54f711983 is not a gfid? What else is it? > >Mathias > >On 08.08.20 09:00, Strahil Nikolov wrote: >> In glusterfs the long string is called "gfid" and does not represent >the name. >> >> Best Regards, >> Strahil Nikolov >> >> >> >> >> >> >> В петък, 7 август 2020 г., 21:40:11 Гринуич+3, Mathias Waack ><mathias.waack@xxxxxxxxxxxxxxx> написа: >> >> >> >> >> >> Hi Strahil, >> >> but I cannot find these files in the heal info: >> >> find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> ' >> ... >> 7443397 132463 -rw------- 1 999 docker 1073741824 Aug 3 >10:35 >> /zbrick/.glusterfs/b5/3c/b53c8e46-068b-4286-94a6-7cf54f711983 >> >> Now looking for this file in the heal infos: >> >> gluster volume heal gvol info | grep >b53c8e46-068b-4286-94a6-7cf54f711983 >> >> shows nothing. >> >> So I do not know, what I have to heal... >> >> Mathias >> >> On 07.08.20 14:32, Strahil Nikolov wrote: >>> Have you tried to gluster heal and check if the files are back into >their place? >>> >>> I always thought that those hard links are used by the healing >mechanism and if that is true - gluster should restore the files to >their original location and then wiping the correct files from FUSE >will be easy. >>> >>> Best Regards, >>> Strahil Nikolov >>> >>> На 7 август 2020 г. 10:24:38 GMT+03:00, Mathias Waack ><mathias.waack@xxxxxxxxxxxxxxx> написа: >>>> Hi all, >>>> >>>> maybe I should add some more information: >>>> >>>> The container which filled up the space was running on node x, >which >>>> still shows a nearly filled fs: >>>> >>>> 192.168.1.x:/gvol 2.6T 2.5T 149G 95% /gluster >>>> >>>> nearly the same situation on the underlying brick partition on node >x: >>>> >>>> zdata/brick 2.6T 2.4T 176G 94% /zbrick >>>> >>>> On node y the network card crashed, glusterfs shows the same >values: >>>> >>>> 192.168.1.y:/gvol 2.6T 2.5T 149G 95% /gluster >>>> >>>> but different values on the brick: >>>> >>>> zdata/brick 2.9T 1.6T 1.4T 54% /zbrick >>>> >>>> I think this happened because glusterfs still has hardlinks to the >>>> deleted files on node x? So I can find these files with: >>>> >>>> find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> ' >>>> >>>> But now I am lost. How can I verify these files really belongs to >the >>>> right container? Or can I just delete this files because there is >no >>>> way >>>> to access it? Or offers glusterfs a way to solve this situation? >>>> >>>> Mathias >>>> >>>> On 05.08.20 15:48, Mathias Waack wrote: >>>>> Hi all, >>>>> >>>>> we are running a gluster setup with two nodes: >>>>> >>>>> Status of volume: gvol >>>>> Gluster process TCP Port RDMA Port >>>>> Online Pid >>>>> >>>> >------------------------------------------------------------------------------ >>>> >>>>> Brick 192.168.1.x:/zbrick 49152 0 Y 13350 >>>>> Brick 192.168.1.y:/zbrick 49152 0 Y 5965 >>>>> Self-heal Daemon on localhost N/A N/A Y 14188 >>>>> Self-heal Daemon on 192.168.1.93 N/A N/A Y 6003 >>>>> >>>>> Task Status of Volume gvol >>>>> >>>> >------------------------------------------------------------------------------ >>>> >>>>> There are no active volume tasks >>>>> >>>>> The glusterfs hosts a bunch of containers with its data volumes. >The >>>>> underlying fs is zfs. Few days ago one of the containers created a >>>> lot >>>>> of files in one of its data volumes, and at the end it completely >>>>> filled up the space of the glusterfs volume. But this happened >only >>>> on >>>>> one host, on the other host there was still enough space. We >finally >>>>> were able to identify this container and found out, the sizes of >the >>>>> data on /zbrick were different on both hosts for this container. >Now >>>>> we made the big mistake to delete these files on both hosts in the >>>>> /zbrick volume, not on the mounted glusterfs volume. >>>>> >>>>> Later we found the reason for this behavior: the network driver on >>>> the >>>>> second node partially crashed (which means we ware able to login >on >>>>> the node, so we assumed the network was running, but the card was >>>>> already dropping packets at this time) at the same time, as the >>>> failed >>>>> container started to fill up the gluster volume. After rebooting >the >>>>> second node the gluster became available again. >>>>> >>>>> Now the glusterfs volume is running again- but it is still >(nearly) >>>>> full: the files created by the container are not visible, but they >>>>> still count into amount of free space. How can we fix this? >>>>> >>>>> In addition there are some files which are no longer accessible >since >>>>> this accident: >>>>> >>>>> tail access.log.old >>>>> tail: cannot open 'access.log.old' for reading: Input/output error >>>>> >>>>> Looks like affected by this error are files which have been >changed >>>>> during the accident. Is there a way to fix this too? >>>>> >>>>> Thanks >>>>> Mathias >>>>> >>>>> >>>>> ________ >>>>> >>>>> >>>>> >>>>> Community Meeting Calendar: >>>>> >>>>> Schedule - >>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >>>>> Bridge: https://bluejeans.com/441850968 >>>>> >>>>> Gluster-users mailing list >>>>> Gluster-users@xxxxxxxxxxx >>>>> https://lists.gluster.org/mailman/listinfo/gluster-users >>>> ________ >>>> >>>> >>>> >>>> Community Meeting Calendar: >>>> >>>> Schedule - >>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >>>> Bridge: https://bluejeans.com/441850968 >>>> >>>> Gluster-users mailing list >>>> Gluster-users@xxxxxxxxxxx >>>> https://lists.gluster.org/mailman/listinfo/gluster-users >> ________ >> >> >> >> Community Meeting Calendar: >> >> Schedule - >> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >> Bridge: https://bluejeans.com/441850968 >> >> Gluster-users mailing list >> Gluster-users@xxxxxxxxxxx >> https://lists.gluster.org/mailman/listinfo/gluster-users >________ > > > >Community Meeting Calendar: > >Schedule - >Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC >Bridge: https://bluejeans.com/441850968 > >Gluster-users mailing list >Gluster-users@xxxxxxxxxxx >https://lists.gluster.org/mailman/listinfo/gluster-users ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://bluejeans.com/441850968 Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
