Hi Jeff. Thanks for the quick response. You can find the output here: http://pastebin.com/PVzBTSxq I should note that the mirror pair jc1letgfs17 and 18 were created first, and then the pair jc1letgfs14 and 15 were added using the gluster volume add-brick command below: gluster volume add-brick pfs-ro1 jc1letgfs14-pfs1:/export/read-only/g01 jc1letgfs15-pfs1:/export/read-only/g01 jc1letgfs14-pfs1:/export/read-only/g02 jc1letgfs15-pfs1:/export/read-only/g02 jc1letgfs14-pfs1:/export/read-only/g03 jc1letgfs15-pfs1:/export/read-only/g03 jc1letgfs14-pfs1:/export/read-only/g04 jc1letgfs15-pfs1:/export/read-only/g04 jc1letgfs14-pfs1:/export/read-only/g05 jc1letgfs15-pfs1:/export/read-only/g05 jc1letgfs14-pfs1:/export/read-only/g06 jc1letgfs15-pfs1:/export/read-only/g06 jc1letgfs14-pfs1:/export/read-only/g07 jc1letgfs15-pfs1:/export/read-only/g07 jc1letgfs14-pfs1:/export/read-only/g08 jc1letgfs15-pfs1:/export/read-only/g08 jc1letgfs14-pfs1:/export/read-only/g09 jc1letgfs15-pfs1:/export/read-only/g09 jc1letgfs14-pfs1:/export/read-only/g10 jc1letgfs15-pfs1:/export/read-only/g10 Thanks, James Burnash, Unix Engineering -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Jeff Darcy Sent: Monday, March 28, 2011 12:53 PM To: gluster-users at gluster.org Subject: Re: strange error hangs hangs any access to gluster mount On 03/28/2011 12:43 PM, Burnash, James wrote: > I am receiving an error on a client trying to access a gluster mount > (/pfs2, in this case). > > [2011-03-28 12:26:17.897887] I > [dht-layout.c:588:dht_layout_normalize] pfs-ro1-dht: found anomalies > in /. holes=1 overlaps=2 > > This is seen on the client in the /var/log/glusterfs/pfs2.log, which > is the mount point associated with that storage. > > All other clients accessing the same storage do not have the hanging > symptom, and have no such entry in their logs. > > One possibly helpful note - this node worked fine until I upgraded > the client from 3.1.1-1 to 3.1.3-1 on the x86_64 architecture, > running CentOS 5.2. Even after I completely uninstalled GlusterFS > from this node and reinstalled 3.1.1-1, the problem persisted. > > Here is the RPM info: > > root at jc1lnxsamm33:~# rpm -qa fuse fuse-2.7.4-8.el5.x86_64 > root at jc1lnxsamm33:~# rpm -qa "glusterfs*" > glusterfs-fuse-3.1.1-1.x86_64 glusterfs-core-3.1.1-1.x86_64 > glusterfs-debuginfo-3.1.1-1.x86_64 > > Servers are 4 Replicated-Distribute machines running CentOS 5.5 and > GlusterFs 3.1.3-1. > > Volume Name: pfs-ro1 Type: Distributed-Replicate Status: Started > Number of Bricks: 20 x 2 = 40 Transport-type: tcp Bricks: Brick1: > jc1letgfs17-pfs1:/export/read-only/g01 Brick2: > jc1letgfs18-pfs1:/export/read-only/g01 Brick3: > jc1letgfs17-pfs1:/export/read-only/g02 Brick4: > jc1letgfs18-pfs1:/export/read-only/g02 ... Brick35: > jc1letgfs14-pfs1:/export/read-only/g08 Brick36: > jc1letgfs15-pfs1:/export/read-only/g08 Brick37: > jc1letgfs14-pfs1:/export/read-only/g09 Brick38: > jc1letgfs15-pfs1:/export/read-only/g09 Brick39: > jc1letgfs14-pfs1:/export/read-only/g10 Brick40: > jc1letgfs15-pfs1:/export/read-only/g10 Options Reconfigured: > performance.stat-prefetch: on performance.cache-size: 2GB > network.ping-timeout: 10 > > Any help greatly appreciated. Can you execute the following command on each of the brick roots? getfattr -d -e hex -n trusted.glusterfs.dht $brick_root That should give a clearer picture of what the layouts look like, and what those gaps/overlaps are. How they happened is a bit of another story. I see this kind of thing pretty often, but I know it's because of some Weird Stuff (tm) I do in CloudFS. I'm not aware of any bugs etc. that would cause this in other contexts. _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com
