rsync causing gluster to crash

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello Vikas,

Thank you for your help.  Gluster does not seem to core dump when this
occurrence happens thus it is not creating a dump file.  Hopefully I can
provide all the information you need to help you diagnose this issue.

Per the original email we get an error message indicating a stale NFS
mount.  Unfortunately, we are not using NFS at any point in our setup so
the error is misleading.  Once this condition occurs we cannot connect
to the Gluster mount point without re-mounting (standard umount/mount).

Below are the exact steps to reproduce our particular problem:

---------------------------

/mnt/control <- contains sample (random) files on a LOCAL drive
/mnt/backups <- our glusterfs mount point

---------------------------

--NO gluster
backup0:/mnt# ps auxw | grep glus | grep -v grep

--Mounting Gluster (see bottom for fstab)
backup0:/mnt# mount /mnt/backups/

--Gluster is running
backup0:/mnt# ps auxw | grep glus | grep -v grep
root      4341  0.0  0.0  27844  1456 ?        Ssl  14:29   0:00
/usr/local/sbin/glusterfs --log-level=NORMAL
--volfile=/etc/glusterfs/glusterfs.vol /mnt/backups

--Can browse the dir
backup0:/mnt# cd backups/
backup0:/mnt/backups# ls
10001   100401  100801  101301  10201   102501  103101  103801  104301
105201  105701  106101  106701  107201  107601  10801   108601  10901
109301  1101    110501  11101   111401  11501
1001    100601  101     101401  102201  103001  103401  10401   104401
105301  106001  106401  106801  107301  107701  108101  108701  109101
109401  110101  110801  111101  111701  115101
100101  100701  10101   102001  102401  10301   103501  104201  10501
105601  10601   106501  10701   107501  107901  108401  108901  109201
11001   110201  111001  111301  111901  115701

--Run Rsync
backup0:/mnt# rsync -rav -X /mnt/control/ /mnt/backups/ 2> output.txt 1>
output.txt

--Nothing on this machine is NFS... weird error
backup0:/mnt# cd backups
  bash: cd: backups: Stale NFS file handle

--Gluster is still running
backup0:/mnt# ps auxw |grep glu | grep -v grep | grep -v tail
root      4780  0.4  0.0  29900  2480 ?        Ssl  14:42   0:01
/usr/local/sbin/glusterfs --log-level=NORMAL
--volfile=/etc/glusterfs/glusterfs.vol /mnt/backups

--Our version of Gluster
backup0:/usr/local/src# ls -al /usr/local/src/glusterfs-3.0.3.tar.gz
-rw-r--r-- 1 root staff 1684933 2010-03-14 00:04
/usr/local/src/glusterfs-3.0.3.tar.gz

---------------------------

backup0:/mnt# cat /etc/fstab
# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    defaults        0       0
/dev/sda1       /               ext3    errors=remount-ro 0       1
/dev/sdb1       /mnt/sdb1       ext3    errors=remount-ro 0       1
/dev/sdc1       /mnt/sdc1       ext3    errors=remount-ro 0       1
/dev/sdd1       /mnt/sdd1       ext3    errors=remount-ro 0       1
/dev/sda5       none            swap    sw              0       0
/etc/glusterfs/glusterfs.vol  /mnt/backups  glusterfs  defaults  0  0

---------------------------

backup0:/mnt# cat /etc/glusterfs/glusterfs.vol
### file: client-volume.vol.sample

#####################################
###  GlusterFS Client Volume File  ##
#####################################

#### CONFIG FILE RULES:
### "#" is comment character.
### - Config file is case sensitive
### - Options within a volume block can be in any order.
### - Spaces or tabs are used as delimitter within a line.
### - Each option should end within a line.
### - Missing or commented fields will assume default values.
### - Blank/commented lines are allowed.
### - Sub-volumes should already be defined above before referring.

# drives
volume backup1-sda1
  type protocol/client
  option transport-type tcp
  option remote-host 10.10.0.71
  option remote-subvolume sda1
end-volume

volume backup2-sda1
  type protocol/client
  option transport-type tcp
  option remote-host 10.10.0.72
  option remote-subvolume sda1
end-volume

volume backup3-sda1
  type protocol/client
  option transport-type tcp
  option remote-host 10.10.0.73
  option remote-subvolume sda1
end-volume

volume backup4-sda1
  type protocol/client
  option transport-type tcp
  option remote-host 10.10.0.74
  option remote-subvolume sda1
end-volume

# replication
volume b-1-2-sda1
  type cluster/replicate
  subvolumes backup1-sda1 backup2-sda1
end-volume

volume b-3-4-sda1
  type cluster/replicate
  subvolumes backup3-sda1 backup4-sda1
end-volume

# striping
  volume distribute
  type cluster/distribute
  subvolumes b-1-2-sda1 b-3-4-sda1
end-volume

-------------------------------------

Gluster Log

========================================================================
========
Version      : glusterfs 3.0.3 built on Mar 19 2010 11:55:08
git: v3.0.2-41-g029062c
Starting Time: 2010-03-19 14:50:34
Command line : /usr/local/sbin/glusterfs --log-level=NORMAL
--volfile=/etc/glusterfs/glusterfs.vol /mnt/backups
PID          : 4941
System name  : Linux
Nodename     : backup0
Kernel Release : 2.6.30-2-686
Hardware Identifier: i686

Given volfile:
+-----------------------------------------------------------------------
-------+
  1: ### file: client-volume.vol.sample
  2:
  3: #####################################
  4: ###  GlusterFS Client Volume File  ##
  5: #####################################
  6:
  7: #### CONFIG FILE RULES:
  8: ### "#" is comment character.
  9: ### - Config file is case sensitive
 10: ### - Options within a volume block can be in any order.
 11: ### - Spaces or tabs are used as delimitter within a line.
 12: ### - Each option should end within a line.
 13: ### - Missing or commented fields will assume default values.
 14: ### - Blank/commented lines are allowed.
 15: ### - Sub-volumes should already be defined above before referring.
 16:
 17: # drives
 18: volume backup1-sda1
 19:   type protocol/client
 20:   option transport-type tcp
 21:   option remote-host 10.10.0.71
 22:   option remote-subvolume sda1
 23: end-volume
 24:
 25: volume backup2-sda1
 26:   type protocol/client
 27:   option transport-type tcp
 28:   option remote-host 10.10.0.72
 29:   option remote-subvolume sda1
 30: end-volume
 31:
 32: volume backup3-sda1
 33:   type protocol/client
 34:   option transport-type tcp
 35:   option remote-host 10.10.0.73
 36:   option remote-subvolume sda1
 37: end-volume
 38:
 39: volume backup4-sda1
 40:   type protocol/client
 41:   option transport-type tcp
 42:   option remote-host 10.10.0.74
 43:   option remote-subvolume sda1
 44: end-volume
 45:
 46: # replication
 47: volume b-1-2-sda1
 48:   type cluster/replicate
 49:   subvolumes backup1-sda1 backup2-sda1
 50: end-volume
 51:
 52: volume b-3-4-sda1
 53:   type cluster/replicate
 54:   subvolumes backup3-sda1 backup4-sda1
 55: end-volume
 56:
 57: # striping
 58: volume distribute
 59:   type cluster/distribute
 60:   subvolumes b-1-2-sda1 b-3-4-sda1
 61: end-volume
62:
 63: #### Add readahead feature
 64: #volume readahead
 65: #  type performance/read-ahead
 66: #  option page-size 1MB     # unit in bytes
 67: #  option page-count 2      # cache per file  = (page-count x
page-size)
 68: #  subvolumes distribute
 69: #end-volume
 70: #
 71: #### Add IO-Cache feature
 72: #volume iocache
 73: #    type performance/io-cache
 74: #    option cache-size `grep 'MemTotal' /proc/meminfo  | awk
'{print $2 * 0.2 / 1024}' | cut -f1 -d.`MB
 75: #    option cache-timeout 1
 76: #    subvolumes readahead
 77: #end-volume
 78: #
 79: #### Add quick-read feature
 80: #volume quickread
 81: #    type performance/quick-read
 82: #    option cache-timeout 1
 83: #    option max-file-size 64kB
 84: #    subvolumes iocache
 85: #end-volume
 86: #
 87: #### Add writeback feature
 88: #volume writeback
 89: #  type performance/write-behind
 90: #  option aggregate-size 1MB
 91: #  option window-size 2MB
 92: #  option flush-behind off
 93: #  subvolumes quickread
 94: #end-volume
 95: #

+-----------------------------------------------------------------------
-------+
[2010-03-19 14:50:34] N [glusterfsd.c:1396:main] glusterfs: Successfully
started
[2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk]
backup4-sda1: Connected to 10.10.0.74:6996, attached to remote volume
'sda1'.
[2010-03-19 14:50:34] N [afr.c:2627:notify] b-3-4-sda1: Subvolume
'backup4-sda1' came back up; going online.
[2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk]
backup4-sda1: Connected to 10.10.0.74:6996, attached to remote volume
'sda1'.
[2010-03-19 14:50:34] N [afr.c:2627:notify] b-3-4-sda1: Subvolume
'backup4-sda1' came back up; going online.
[2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk]
backup2-sda1: Connected to 10.10.0.72:6996, attached to remote volume
'sda1'.
[2010-03-19 14:50:34] N [afr.c:2627:notify] b-1-2-sda1: Subvolume
'backup2-sda1' came back up; going online.
[2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk]
backup2-sda1: Connected to 10.10.0.72:6996, attached to remote volume
'sda1'.
[2010-03-19 14:50:34] N [afr.c:2627:notify] b-1-2-sda1: Subvolume
'backup2-sda1' came back up; going online.
[2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk]
backup1-sda1: Connected to 10.10.0.71:6996, attached to remote volume
'sda1'.
[2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk]
backup1-sda1: Connected to 10.10.0.71:6996, attached to remote volume
'sda1'.
[2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk]
backup3-sda1: Connected to 10.10.0.73:6996, attached to remote volume
'sda1'.
[2010-03-19 14:50:34] N [client-protocol.c:6246:client_setvolume_cbk]
backup3-sda1: Connected to 10.10.0.73:6996, attached to remote volume
'sda1'.
[2010-03-19 14:50:34] N [fuse-bridge.c:2942:fuse_init] glusterfs-fuse:
FUSE inited with protocol versions: glusterfs 7.13 kernel 7.11
[2010-03-19 14:50:49] W [fuse-bridge.c:491:fuse_entry_cbk]
glusterfs-fuse: LOOKUP(/100101) inode (ptr=0x9a57280, ino=373293060,
gen=5450392427238002552) found conflict (ptr=0x9a509a8, ino=373293060,
gen=5450392427238002552)
[2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk]
glusterfs-fuse: 1349: LOOKUP() / => -1 (Stale NFS file handle)
[2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk]
glusterfs-fuse: 1350: LOOKUP() / => -1 (Stale NFS file handle)
[2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk]
glusterfs-fuse: 1351: LOOKUP() / => -1 (Stale NFS file handle)
[2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk]
glusterfs-fuse: 1352: LOOKUP() / => -1 (Stale NFS file handle)
[2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk]
glusterfs-fuse: 1353: LOOKUP() / => -1 (Stale NFS file handle)
[2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk]
glusterfs-fuse: 1354: LOOKUP() / => -1 (Stale NFS file handle)
[2010-03-19 14:50:49] W [fuse-bridge.c:722:fuse_attr_cbk]
glusterfs-fuse: 1355: LOOKUP() / => -1 (Stale NFS file handle)
[Repeated for hundreds of lines]



----------------------------------
Benjamin Hudgens
Director of Information Technology
Photodex Corporation

www.photodex.com
bhudgens at photodex.com
(512) 674-9920




-----Original Message-----
From: gluster-users-bounces at gluster.org
[mailto:gluster-users-bounces at gluster.org] On Behalf Of Vikas Gorur
Sent: Friday, March 19, 2010 2:02 PM
To: Joe Grace
Cc: Gluster Users
Subject: Re: rsync causing gluster to crash


On Mar 19, 2010, at 9:47 AM, Joe Grace wrote:

> Thank you for the reply.
> 
> Version 3.0.2 from source on Debian sqeeze.
> 
> Here is the client log:


That is the client volume file. What I meant was the client log file you
can usually find in /usr/local/var/log/glusterfs/glusterfs.log.

If you have a core file from glusterfs (look in /core.*), you can get a
backtrace by:

# gdb /path/to/glusterfs /path/to/core/file
(gdb) thread apply all bt full

Please send us the output of that command.

------------------------------
Vikas Gorur
Engineer - Gluster, Inc.
+1 (408) 770 1894
------------------------------







_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux