Oh I see, I got you wrong. Now I am going to start to understand the
whole thing. Thank you for the comprehensive explanation. For good luck
it is weekend, so I can start digging into this...
Thanks
Mathias
On 08.08.20 18:05, Strahil Nikolov wrote:
If you read my previous email, you will see that i noted that the string IS GFID and not the name of the file :)
You can find the name following the procedure at: https://docs.gluster.org/en/latest/Troubleshooting/gfid-to-path/
Of course, that will be slow for all entries in .glusterfs and you will need to create a script to match all gfids to brick path.
I guess the fastest way to find the deleted files (As far as I understood they were deleted on the brick directly and entries in .glusterfs were left) is to create a script that:
0.Create a ramfs for the files:
findmnt /mnt || mount -t ramfs -o size=128MB - /mnt
1. Get all inodes
ionice -c 2 -n 7 nice -n 15 find /full/path/to/brick -type f -exec ls -i {} \; >/mnt/data
2. Get only the inodes:
nice -n 15 awk '{print $1}' /mnt/data > /mnt/inode_only
3. Now the fun starts now-> find inodes that are not duplicate:
nice -n 15 uniq -u /mnt/inode_only > /mnt/gfid-only
4. Once you have the inodes, you can verify that they do exists only in .gluster dir
for i in $(cat /mnt/gfid-only); do ionice -c 2 -n 7 nice -n 15 find /path/to/.glusterfs -inum $i ; echo;echo; done
5. If it's OK -> delete
for i in $(cat /mnt/gfid-only); do ionice -c 2 -n 7 nice -n 15 find /path/to/brick -inum $i -delete ; done
Last , repeat on all bricks
Good luck!
P.S.: Consider creating a gluster snapshot before that - just in case... Better safe than sorry.
P.S: If you think that you got enough resources, you can remove ionice/nice stuff . They are just to guarantee you won't eat too many resources.
Best Regards,
Strahil Nikolov
На 8 август 2020 г. 18:02:10 GMT+03:00, Mathias Waack <mathias.waack@xxxxxxxxxxxxxxx> написа:
So b53c8e46-068b-4286-94a6-7cf54f711983 is not a gfid? What else is it?
Mathias
On 08.08.20 09:00, Strahil Nikolov wrote:
In glusterfs the long string is called "gfid" and does not represent
the name.
Best Regards,
Strahil Nikolov
В петък, 7 август 2020 г., 21:40:11 Гринуич+3, Mathias Waack
<mathias.waack@xxxxxxxxxxxxxxx> написа:
Hi Strahil,
but I cannot find these files in the heal info:
find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
...
7443397 132463 -rw------- 1 999 docker 1073741824 Aug 3
10:35
/zbrick/.glusterfs/b5/3c/b53c8e46-068b-4286-94a6-7cf54f711983
Now looking for this file in the heal infos:
gluster volume heal gvol info | grep
b53c8e46-068b-4286-94a6-7cf54f711983
shows nothing.
So I do not know, what I have to heal...
Mathias
On 07.08.20 14:32, Strahil Nikolov wrote:
Have you tried to gluster heal and check if the files are back into
their place?
I always thought that those hard links are used by the healing
mechanism and if that is true - gluster should restore the files to
their original location and then wiping the correct files from FUSE
will be easy.
Best Regards,
Strahil Nikolov
На 7 август 2020 г. 10:24:38 GMT+03:00, Mathias Waack
<mathias.waack@xxxxxxxxxxxxxxx> написа:
Hi all,
maybe I should add some more information:
The container which filled up the space was running on node x,
which
still shows a nearly filled fs:
192.168.1.x:/gvol 2.6T 2.5T 149G 95% /gluster
nearly the same situation on the underlying brick partition on node
x:
zdata/brick 2.6T 2.4T 176G 94% /zbrick
On node y the network card crashed, glusterfs shows the same
values:
192.168.1.y:/gvol 2.6T 2.5T 149G 95% /gluster
but different values on the brick:
zdata/brick 2.9T 1.6T 1.4T 54% /zbrick
I think this happened because glusterfs still has hardlinks to the
deleted files on node x? So I can find these files with:
find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
But now I am lost. How can I verify these files really belongs to
the
right container? Or can I just delete this files because there is
no
way
to access it? Or offers glusterfs a way to solve this situation?
Mathias
On 05.08.20 15:48, Mathias Waack wrote:
Hi all,
we are running a gluster setup with two nodes:
Status of volume: gvol
Gluster process TCP Port RDMA Port
Online Pid
------------------------------------------------------------------------------
Brick 192.168.1.x:/zbrick 49152 0 Y 13350
Brick 192.168.1.y:/zbrick 49152 0 Y 5965
Self-heal Daemon on localhost N/A N/A Y 14188
Self-heal Daemon on 192.168.1.93 N/A N/A Y 6003
Task Status of Volume gvol
------------------------------------------------------------------------------
There are no active volume tasks
The glusterfs hosts a bunch of containers with its data volumes.
The
underlying fs is zfs. Few days ago one of the containers created a
lot
of files in one of its data volumes, and at the end it completely
filled up the space of the glusterfs volume. But this happened
only
on
one host, on the other host there was still enough space. We
finally
were able to identify this container and found out, the sizes of
the
data on /zbrick were different on both hosts for this container.
Now
we made the big mistake to delete these files on both hosts in the
/zbrick volume, not on the mounted glusterfs volume.
Later we found the reason for this behavior: the network driver on
the
second node partially crashed (which means we ware able to login
on
the node, so we assumed the network was running, but the card was
already dropping packets at this time) at the same time, as the
failed
container started to fill up the gluster volume. After rebooting
the
second node the gluster became available again.
Now the glusterfs volume is running again- but it is still
(nearly)
full: the files created by the container are not visible, but they
still count into amount of free space. How can we fix this?
In addition there are some files which are no longer accessible
since
this accident:
tail access.log.old
tail: cannot open 'access.log.old' for reading: Input/output error
Looks like affected by this error are files which have been
changed
during the accident. Is there a way to fix this too?
Thanks
Mathias
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
