Hi Gluster Gurus, I'm using a gluster volume as home for our users. The volume is replica 3, running on CentOS 7, gluster version 3.10 (3.10.6-1.el7.x86_64). Clients are running Fedora 26 and also gluster 3.10 (3.10.6-3.fc26.x86_64). During the data backup I got an I/O error on one file. Manually checking for this file on a client confirms this: ls -l romanoch/.mozilla/firefox/vzzqqxrm.default-1396429081309/sessionstore-backups/ ls: cannot access 'romanoch/.mozilla/firefox/vzzqqxrm.default-1396429081309/sessionstore-backups/recovery.baklz4': Input/output error total 2015 -rw-------. 1 romanoch tbi 998211 Sep 15 18:44 previous.js -rw-------. 1 romanoch tbi 65222 Oct 17 17:57 previous.jsonlz4 -rw-------. 1 romanoch tbi 149161 Oct 1 13:46 recovery.bak -?????????? ? ? ? ? ? recovery.baklz4 Out of curiosity I checked all the bricks for this file. It's present there. Making a checksum shows that the file is different on one of the three replica servers. Querying healing information shows that the file should be healed: # gluster volume heal home info Brick sphere-six:/srv/gluster_home/brick /romanoch/.mozilla/firefox/vzzqqxrm.default-1396429081309/sessionstore-backups/recovery.baklz4 Status: Connected Number of entries: 1 Brick sphere-five:/srv/gluster_home/brick /romanoch/.mozilla/firefox/vzzqqxrm.default-1396429081309/sessionstore-backups/recovery.baklz4 Status: Connected Number of entries: 1 Brick sphere-four:/srv/gluster_home/brick Status: Connected Number of entries: 0 Manually triggering heal doesn't report an error but also does not heal the file. # gluster volume heal home Launching heal operation to perform index self heal on volume home has been successful Same with a full heal # gluster volume heal home full Launching heal operation to perform full self heal on volume home has been successful According to the split brain query that's not the problem: # gluster volume heal home info split-brain Brick sphere-six:/srv/gluster_home/brick Status: Connected Number of entries in split-brain: 0 Brick sphere-five:/srv/gluster_home/brick Status: Connected Number of entries in split-brain: 0 Brick sphere-four:/srv/gluster_home/brick Status: Connected Number of entries in split-brain: 0 I have no idea why this situation arose in the first place and also no idea as how to solve this problem. I would highly appreciate any helpful feedback I can get. The only mention in the logs matching this file is a rename operation: /var/log/glusterfs/bricks/srv-gluster_home-brick.log:[2017-10-23 09:19:11.561661] I [MSGID: 115061] [server-rpc-fops.c:1022:server_rename_cbk] 0-home-server: 5266153: RENAME /romanoch/.mozilla/firefox/vzzqqxrm.default-1396429081309/sessionstore-backups/recovery.jsonlz4 (48e9eea6-cda6-4e53-bb4a-72059debf4c2/recovery.jsonlz4) -> /romanoch/.mozilla/firefox/vzzqqxrm.default-1396429081309/sessionstore-backups/recovery.baklz4 (48e9eea6-cda6-4e53-bb4a-72059debf4c2/recovery.baklz4), client: romulus.tbi.univie.ac.at-11894-2017/10/18-07:06:07:206366-home-client-3-0-0, error-xlator: home-posix [No data available] I enabled directory quotas the same day this problem showed up but I'm not sure how quotas could have an effect like this (maybe unless the limit is reached but that's also not the case). Thanks again if anyone as an idea. Cheers Richard -- /dev/null
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
