Would apprecaite any insight into this issue:
replica 3 volume, it is showing a number of files on two of the bricks as needing healed, when you examine the files on the fuse mounts they generate I/O errors.
No files listed in split brain, but if I look at one of the files it looks to me like they have been updated on gluster-2 and gluster0 but not on gluster1 (see below).replica 3 volume, it is showing a number of files on two of the bricks as needing healed, when you examine the files on the fuse mounts they generate I/O errors.
I see errors in /va/log/gluster/glustershd.log
-Thanks Alastair
[2016-12-20 07:25:06.018829] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-12-20 07:25:06.018901] E [socket.c:2309:socket_connect_finish] 0-glusterfs: connection to ::1:24007 failed (Connection refused)
[2016-12-20 07:25:06.018944] E [glusterfsd-mgmt.c:1902:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: localhost (Transport endpoint is not connected)
[2016-12-20 07:25:07.187710] W [glusterfsd.c:1327:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fd93f669dc5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fd940cfbcd5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fd940cfbb4b] ) 0-: received signum (15), shutting down
[2016-12-20 07:25:08.197959] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.5 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/3fe0b238bd46c38a95636f25cb5b9d8a.socket --xlator-option *replicate*.node-uuid=bcff5245-ea86-4384-a1bf-9219c8be8001)
[2016-12-20 07:25:08.216336] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-12-20 07:25:08.216419] E [socket.c:2309:socket_connect_finish] 0-glusterfs: connection to ::1:24007 failed (Connection refused)
[2016-12-20 07:25:08.216464] E [glusterfsd-mgmt.c:1902:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: localhost (Transport endpoint is not connected)
[2016-12-20 07:25:12.208092] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-digitalcorpora-replicate-0: adding option 'node-uuid' for volume 'digitalcorpora-replicate-0' with value 'bcff5245-ea86-4384-a1bf-9219c8be8001'
[2016-12-20 07:25:12.208122] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-gluster_shared_storage-replicate-0: adding option 'node-uuid' for volume 'gluster_shared_storage-replicate-0' with value 'bcff5245-ea86-4384-a1bf-9219c8be8001'
[2016-12-20 07:25:12.208140] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-homes-replicate-0: adding option 'node-uuid' for volume 'homes-replicate-0' with value 'bcff5245-ea86-4384-a1bf-9219c8be8001'
[2016-12-20 07:25:12.208155] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-public-replicate-0: adding option 'node-uuid' for volume 'public-replicate-0' with value 'bcff5245-ea86-4384-a1bf-9219c8be8001'
[2016-12-20 07:25:12.208173] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-static-web-replicate-0: adding option 'node-uuid' for volume 'static-web-replicate-0' with value 'bcff5245-ea86-4384-a1bf-9219c8be8001'
[2016-12-20 07:25:12.208199] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-tmp-replicate-0: adding option 'node-uuid' for volume 'tmp-replicate-0' with value 'bcff5245-ea86-4384-a1bf-9219c8be8001'
[2016-12-20 07:25:12.208215] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-usr-local-replicate-0: adding option 'node-uuid' for volume 'usr-local-replicate-0' with value 'bcff5245-ea86-4384-a1bf-9219c8be8001'
[2016-12-20 18:32:06.121734] E [client-common.c:526:client_pre_getxattr] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xb5d8) [0x7f6bc4ba65d8] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x26ebd) [0x7f6bc4bc1ebd] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x393e3) [0x7f6bc4bd43e3] ) 0-: Assertion failed: 0
[2016-12-20 18:32:06.121809] E [client-common.c:587:client_pre_opendir] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xa9d5) [0x7f6bc4ba59d5] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x25a65) [0x7f6bc4bc0a65] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x396b7) [0x7f6bc4bd46b7] ) 0-: Assertion failed: 0
[2016-12-20 18:46:51.764776] E [client-common.c:526:client_pre_getxattr] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xb5d8) [0x7f6bc4ba65d8] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x26ebd) [0x7f6bc4bc1ebd] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x393e3) [0x7f6bc4bd43e3] ) 0-: Assertion failed: 0
[2016-12-20 18:46:51.764850] E [client-common.c:587:client_pre_opendir] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xa9d5) [0x7f6bc4ba59d5] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x25a65) [0x7f6bc4bc0a65] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x396b7) [0x7f6bc4bd46b7] ) 0-: Assertion failed: 0
[2016-12-20 18:49:29.657568] E [client-common.c:526:client_pre_getxattr] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xb5d8) [0x7f6bc4ba65d8] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x26ebd) [0x7f6bc4bc1ebd] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x393e3) [0x7f6bc4bd43e3] ) 0-: Assertion failed: 0
[2016-12-20 18:49:29.657645] E [client-common.c:587:client_pre_opendir] (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xa9d5) [0x7f6bc4ba59d5] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x25a65) [0x7f6bc4bc0a65] -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x396b7) [0x7f6bc4bd46b7] ) 0-: Assertion failed: 0
gluster2:
# getfattr -d -m. -e hex /export/brick2/home/a/j/ajn/.Xauthority
getfattr: Removing leading '/' from absolute path names
# file: export/brick2/home/a/j/ajn/.Xauthority
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.homes-client-5=0x000000020000000100000000
trusted.bit-rot.version=0x020000000000000058589e6b0005bdac
trusted.gfid=0xb8b156b764304fd1bf7e692649bcecc5
gluster1:
# getfattr -d -m. -e hex /export/brick2/home/a/j/ajn/.Xauthority
getfattr: Removing leading '/' from absolute path names
# file: export/brick2/home/a/j/ajn/.Xauthority
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x0200000000000000583f45c20008d152
trusted.gfid=0x6c278b5c94ae436bb669b5f5dd21777e
gluster0:
# getfattr -d -m. -e hex /export/brick2/home/a/j/ajn/.Xauthority
getfattr: Removing leading '/' from absolute path names
# file: export/brick2/home/a/j/ajn/.Xauthority
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.homes-client-5=0x000000020000000100000000
trusted.bit-rot.version=0x0200000000000000583f3fbb000b5b01
trusted.gfid=0xb8b156b764304fd1bf7e692649bcecc5
getfattr: Removing leading '/' from absolute path names
# file: export/brick2/home/a/j/ajn/.Xauthority
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.homes-client-5=0x000000020000000100000000
trusted.bit-rot.version=0x0200000000000000583f3fbb000b5b01
trusted.gfid=0xb8b156b764304fd1bf7e692649bcecc5
[root@gluster0 Project3]# glv heal homes info
Brick gluster-2:/export/brick2/home
/s/a/sadams25/pp2.txt
/s/a/sadams25/.viminfo
/a/v/avakil/.Xauthority
/j/m/jmurra17/fork
/c/f/cferris2/.viminfo
/c/s/cs367/bomblab/S001/log-status.txt
/c/s/cs367/bomblab/S001/bomblab-scoreboard.html
/c/s/cs367/bomblab/S001/scores.txt
/c/s/cs367/bomblab/S003/bomblab-scoreboard.html
/c/s/cs367/bomblab/S003/scores.txt
/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/libsupport.a
/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/Makefile
/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.c
/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.h
/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/caching.c
/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.o
/j/m/jmurra17/fork/fork.c
/j/m/jmurra17/.viminfo
/a/j/ajn/.Xauthority
/a/v/avakil/source_code/rm_setup/common_setup.tcl
/a/v/avakil/source_code/rm_setup/dc_setup_filenames.tcl
/a/v/avakil/source_code/rm_setup/dc_setup.tcl
/j/d/jdenton3/.viminfo
/s/a/sadams25/x.txt
/j/d/jdenton3/Project3/Project3.c
/j/m/jmurra17/fork/fork
/j/d/jdenton3/Project3/p5
Status: Connected
Number of entries: 27
Brick gluster1.vsnet.gmu.edu:/export/brick2/home
Status: Connected
Number of entries: 0
Brick gluster0:/export/brick2/home
/s/a/sadams25/pp2.txt
/s/a/sadams25/.viminfo
/c/s/cs367/bomblab/S003/scores.txt
/a/v/avakil/.Xauthority
/c/s/cs367/bomblab/S001/scores.txt
/c/f/cferris2/.viminfo
/c/s/cs367/bomblab/S001/log-status.txt
/c/s/cs367/bomblab/S003/tmpwebpage.14635
/c/s/cs367/bomblab/S001/bomblab-scoreboard.html
/c/s/cs367/bomblab/S003/bomblab-scoreboard.html
/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/libsupport.a
/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/Makefile
/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.c
/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.h
/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/caching.c
/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.o
/j/m/jmurra17/fork
<gfid:310211c2-aeec-4906-894f-023d0ad7d5cc>/#affiliate.nagios.com/settings.sol
/a/v/avakil/source_code/rm_setup/common_setup.tcl
/a/j/ajn/.Xauthority
/j/m/jmurra17/.viminfo
/a/v/avakil/source_code/rm_setup/dc_setup.tcl
/j/m/jmurra17/fork/fork.c
/a/v/avakil/source_code/rm_setup/dc_setup_filenames.tcl
/j/d/jdenton3/Project3/Project3.c
/j/d/jdenton3/.viminfo
/s/a/sadams25/x.txt
/j/m/jmurra17/fork/fork
/j/d/jdenton3/Project3/p5
Status: Connected
Number of entries: 29
[
[root@gluster0 .bad]# cd /mnt/home/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/
[root@gluster0 mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh]# ls -al
ls: cannot access libsupport.a: Input/output error
ls: cannot access Makefile: Input/output error
ls: cannot access memory_system.c: Input/output error
ls: cannot access memory_system.h: Input/output error
ls: cannot access caching.c: Input/output error
ls: cannot access memory_system.o: Input/output error
total 626
drwxrwxr-x 2 1735 users 4096 Dec 20 11:38 .
drwxr-xr-x 3 root root 4096 Dec 20 13:53 ..
-????????? ? ? ? ? ? caching.c
-rw-rw-r-- 1 1735 users 9056 Dec 20 11:36 caching.o
-rwxrwxr-x 1 1735 users 147855 Dec 20 11:36 lab4
-rw-r--r-- 1 1735 users 307200 Dec 13 07:04 Lab 4 - 12 9_mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh.tar
-rw-rw-r-- 1 1735 users 8254 Dec 20 11:38 lab4_logfile
-rw-r--r-- 1 1735 users 153600 Dec 20 11:32 lab4_mchehreh.tar
-????????? ? ? ? ? ? libsupport.a
-????????? ? ? ? ? ? Makefile
-????????? ? ? ? ? ? memory_system.c
-????????? ? ? ? ? ? memory_system.h
-????????? ? ? ? ? ? memory_system.o
-rw-rw-r-- 1 1735 users 449 Dec 20 11:38 t1
-rw-rw-r-- 1 1735 users 453 Dec 20 11:38 t2
-rw-rw-r-- 1 1735 users 2185 Dec 20 11:38 t3
-rw-rw-r-- 1 1735 users 2195 Dec 20 11:38 t4
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
