Hello, Was there additional information we needed to provide to get help diagnosing our issue with keeping GlusterFS mounted? Kind Regards, Benjamin -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Benjamin Hudgens Sent: Friday, March 19, 2010 2:54 PM Cc: Gluster Users Subject: Re: rsync causing gluster to crash Hello Vikas, Thank you for your help. Gluster does not seem to core dump when this occurrence happens thus it is not creating a dump file. Hopefully I can provide all the information you need to help you diagnose this issue. Per the original email we get an error message indicating a stale NFS mount. Unfortunately, we are not using NFS at any point in our setup so the error is misleading. Once this condition occurs we cannot connect to the Gluster mount point without re-mounting (standard umount/mount). Below are the exact steps to reproduce our particular problem: --------------------------- /mnt/control <- contains sample (random) files on a LOCAL drive /mnt/backups <- our glusterfs mount point --------------------------- --NO gluster backup0:/mnt# ps auxw | grep glus | grep -v grep --Mounting Gluster (see bottom for fstab) backup0:/mnt# mount /mnt/backups/ --Gluster is running backup0:/mnt# ps auxw | grep glus | grep -v grep root 4341 0.0 0.0 27844 1456 ? Ssl 14:29 0:00 /usr/local/sbin/glusterfs --log-level=NORMAL --volfile=/etc/glusterfs/glusterfs.vol /mnt/backups --Can browse the dir backup0:/mnt# cd backups/ backup0:/mnt/backups# ls 10001 100401 100801 101301 10201 102501 103101 103801 104301 105201 105701 106101 106701 107201 107601 10801 108601 10901 109301 1101 110501 11101 111401 11501 1001 100601 101 101401 102201 103001 103401 10401 104401 105301 106001 106401 106801 107301 107701 108101 108701 109101 109401 110101 110801 111101 111701 115101 100101 100701 10101 102001 102401 10301 103501 104201 10501 105601 10601 106501 10701 107501 107901 108401 108901 109201 11001 110201 111001 111301 111901 115701 --Run Rsync backup0:/mnt# rsync -rav -X /mnt/control/ /mnt/backups/ 2> output.txt 1> output.txt --Nothing on this machine is NFS... weird error backup0:/mnt# cd backups bash: cd: backups: Stale NFS file handle --Gluster is still running backup0:/mnt# ps auxw |grep glu | grep -v grep | grep -v tail root 4780 0.4 0.0 29900 2480 ? Ssl 14:42 0:01 /usr/local/sbin/glusterfs --log-level=NORMAL --volfile=/etc/glusterfs/glusterfs.vol /mnt/backups --Our version of Gluster backup0:/usr/local/src# ls -al /usr/local/src/glusterfs-3.0.3.tar.gz -rw-r--r-- 1 root staff 1684933 2010-03-14 00:04 /usr/local/src/glusterfs-3.0.3.tar.gz --------------------------- backup0:/mnt# cat /etc/fstab # /etc/fstab: static file system information. # # <file system> <mount point> <type> <options> <dump> <pass> proc /proc proc defaults 0 0 /dev/sda1 / ext3 errors=remount-ro 0 1 /dev/sdb1 /mnt/sdb1 ext3 errors=remount-ro 0 1 /dev/sdc1 /mnt/sdc1 ext3 errors=remount-ro 0 1 /dev/sdd1 /mnt/sdd1 ext3 errors=remount-ro 0 1 /dev/sda5 none swap sw 0 0 /etc/glusterfs/glusterfs.vol /mnt/backups glusterfs defaults 0 0 --------------------------- backup0:/mnt# cat /etc/glusterfs/glusterfs.vol ### file: client-volume.vol.sample ##################################### ### GlusterFS Client Volume File ## ##################################### #### CONFIG FILE RULES: ### "#" is comment character. ### - Config file is case sensitive ### - Options within a volume block can be in any order. ### - Spaces or tabs are used as delimitter within a line. ### - Each option should end within a line. ### - Missing or commented fields will assume default values. ### - Blank/commented lines are allowed. ### - Sub-volumes should already be defined above before referring. # drives volume backup1-sda1 type protocol/client option transport-type tcp option remote-host 10.10.0.71 option remote-subvolume sda1 end-volume volume backup2-sda1 type protocol/client option transport-type tcp option remote-host 10.10.0.72 option remote-subvolume sda1 end-volume volume backup3-sda1 type protocol/client option transport-type tcp option remote-host 10.10.0.73 option remote-subvolume sda1 end-volume volume backup4-sda1 type protocol/client option transport-type tcp option remote-host 10.10.0.74 option remote-subvolume sda1 end-volume # replication volume b-1-2-sda1 type cluster/replicate subvolumes backup1-sda1 backup2-sda1 end-volume volume b-3-4-sda1 type cluster/replicate subvolumes backup3-sda1 backup4-sda1 end-volume # striping volume distribute type cluster/distribute subvolumes b-1-2-sda1 b-3-4-sda1 end-volume ------------------------------------- Gluster Log ======================================================================== ======== Version : glusterfs 3.0.3 built on Mar 19 2010 11:55:08 git: v3.0.2-41-g029062c Starting Time: 2010-03-19 14:50:34 Command line : /usr/local/sbin/glusterfs --log-level=NORMAL --volfile=/etc/glusterfs/glusterfs.vol /mnt/backups PID : 4941 System name : Linux Nodename : backup0 Kernel Release : 2.6.30-2-686 Hardware Identifier: i686 Given volfile: +----------------------------------------------------------------------- -------+ 1: ### file: client-volume.vol.sample 2: 3: ##################################### 4: ### GlusterFS Client Volume File ## 5: ##################################### 6: 7: #### CONFIG FILE RULES: 8: ### "#" is comment character. 9: ### - Config file is case sensitive 10: ### - Options within a volume block can be in any order. 11: ### - Spaces or tabs are used as delimitter within a line. 12: ### - Each option should end within a line. 13: ### - Missing or commented fields will assume default values. 14: ### - Blank/commented lines are allowed. 15: ### - Sub-volumes should already be defined above before referring. 16: 17: # drives 18: volume backup1-sda1 19: type protocol/client 20: option transport-type tcp 21: option remote-host 10.10.0.71 22: option remote-subvolume sda1 23: end-volume 24: 25: volume backup2-sda1 26: type protocol/client 27: option transport-type tcp 28: option remote-host 10.10.0.72 29: option remote-subvolume sda1 30: end-volume 31: 32: volume backup3-sda1 33: type protocol/client 34: option transport-type tcp 35: option remote-host 10.10.0.73 36: option remote-subvolume sda1 37: end-volume 38: 39: volume backup4-sda1 40: type protocol/client 41: option transport-type tcp 42: option remote-host 10.10.0.74 43: option remote-subvolume sda1 44: end-volume 45: 46: # replication 47: volume b-1-2-sda1 48: type cluster/replicate 49: subvolumes backup1-sda1 backup2-sda1 50: end-volume 51: 52: volume b-3-4-sda1 53: type cluster/replicate 54: subvolumes backup3-sda1 backup4-sda1 55: end-volume 56: 57: # striping 58: volume distribute 59: type cluster/distribute 60: subvolumes b-1-2-sda1 b-3-4-sda1 61: end-volume 62: 63: #### Add readahead feature 64: #volume readahead 65: # type performance/read-ahead 66: # option page-size 1MB # unit in bytes 67: # option page-count 2 # cache per file = (page-count x page-size) 68: # subvolumes distribute 69: #end-volume 70: # 71: #### Add IO-Cache feature 72: #volume iocache 73: # type performance/io-cache 74: # option cache-size `grep 'MemTotal' /proc/meminfo | awk '{print $2 * 0.2 / 1024}' | cut -f1 -d.`MB 75: # option cache-timeout 1 76: # subvolumes readahead 77: #end-volume 78: # 79: #### Add quick-read feature 80: #volume quickread 81: # type performance/quick-read 82: # option cache-timeout 1 83: # option max-file-size 64kB 84: # subvolumes iocache 85: #end-volume 86: # 87: #### Add writeback feature 88: #volume writeback 89: # type performance/write-behind 90: # option aggregate-size 1MB 91: # option window-size 2MB 92: # option flush-behind off 93: # subvolumes quickread 94: #end-volume 95: # +----------------------------------------------------------------------- -------+ [2010-03-19 14:50:34] N [glusterfsd.c:1396:main] glusterfs: Successfully started [2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk] backup4-sda1: Connected to 10.10.0.74:6996, attached to remote volume 'sda1'. [2010-03-19 14:50:34] N [afr.c:2627:notify] b-3-4-sda1: Subvolume 'backup4-sda1' came back up; going online. [2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk] backup4-sda1: Connected to 10.10.0.74:6996, attached to remote volume 'sda1'. [2010-03-19 14:50:34] N [afr.c:2627:notify] b-3-4-sda1: Subvolume 'backup4-sda1' came back up; going online. [2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk] backup2-sda1: Connected to 10.10.0.72:6996, attached to remote volume 'sda1'. [2010-03-19 14:50:34] N [afr.c:2627:notify] b-1-2-sda1: Subvolume 'backup2-sda1' came back up; going online. [2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk] backup2-sda1: Connected to 10.10.0.72:6996, attached to remote volume 'sda1'. [2010-03-19 14:50:34] N [afr.c:2627:notify] b-1-2-sda1: Subvolume 'backup2-sda1' came back up; going online. [2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk] backup1-sda1: Connected to 10.10.0.71:6996, attached to remote volume 'sda1'. [2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk] backup1-sda1: Connected to 10.10.0.71:6996, attached to remote volume 'sda1'. [2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk] backup3-sda1: Connected to 10.10.0.73:6996, attached to remote volume 'sda1'. [2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk] backup3-sda1: Connected to 10.10.0.73:6996, attached to remote volume 'sda1'. [2010-03-19 14:50:34] N [fuse-bridge.c:2942:fuse_init] glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.11 [2010-03-19 14:50:49] W [fuse-bridge.c:491:fuse_entry_cbk] glusterfs-fuse: LOOKUP(/100101) inode (ptr=0x9a57280, ino=373293060, gen=5450392427238002552) found conflict (ptr=0x9a509a8, ino=373293060, gen=5450392427238002552) [2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 1349: LOOKUP() / => -1 (Stale NFS file handle) [2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 1350: LOOKUP() / => -1 (Stale NFS file handle) [2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 1351: LOOKUP() / => -1 (Stale NFS file handle) [2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 1352: LOOKUP() / => -1 (Stale NFS file handle) [2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 1353: LOOKUP() / => -1 (Stale NFS file handle) [2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 1354: LOOKUP() / => -1 (Stale NFS file handle) [2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse: 1355: LOOKUP() / => -1 (Stale NFS file handle) [Repeated for hundreds of lines] ---------------------------------- Benjamin Hudgens Director of Information Technology Photodex Corporation www.photodex.com bhudgens at photodex.com (512) 674-9920 -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Vikas Gorur Sent: Friday, March 19, 2010 2:02 PM To: Joe Grace Cc: Gluster Users Subject: Re: rsync causing gluster to crash On Mar 19, 2010, at 9:47 AM, Joe Grace wrote: > Thank you for the reply. > > Version 3.0.2 from source on Debian sqeeze. > > Here is the client log: That is the client volume file. What I meant was the client log file you can usually find in /usr/local/var/log/glusterfs/glusterfs.log. If you have a core file from glusterfs (look in /core.*), you can get a backtrace by: # gdb /path/to/glusterfs /path/to/core/file (gdb) thread apply all bt full Please send us the output of that command. ------------------------------ Vikas Gorur Engineer - Gluster, Inc. +1 (408) 770 1894 ------------------------------ _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
