Hi Pranith. Yes, I do see those messages in my mount logs on the client: root at jc1lnxsamm100:~# fgrep afr-self-heal /var/log/glusterfs/pfs2.log | tail [2011-06-14 07:30:56.152066] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] 0-pfs-ro1-replicate-6: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-06-14 07:35:16.869848] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] 0-pfs-ro1-replicate-6: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-06-14 07:39:48.500117] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] 0-pfs-ro1-replicate-6: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-06-14 07:40:19.312364] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] 0-pfs-ro1-replicate-6: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-06-14 07:44:27.714292] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] 0-pfs-ro1-replicate-6: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-06-14 07:50:04.691154] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] 0-pfs-ro1-replicate-6: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-06-14 07:54:17.853591] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] 0-pfs-ro1-replicate-6: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-06-14 07:55:26.876415] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] 0-pfs-ro1-replicate-6: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-06-14 07:59:51.702585] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] 0-pfs-ro1-replicate-6: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes [2011-06-14 08:00:08.346056] E [afr-self-heal-metadata.c:524:afr_sh_metadata_fix] 0-pfs-ro1-replicate-6: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes James Burnash Unix Engineer Knight Capital Group -----Original Message----- From: Pranith Kumar. Karampuri [mailto:pranithk at gluster.com] Sent: Tuesday, June 14, 2011 1:28 AM To: Burnash, James; Jeff Darcy (jdarcy at redhat.com); gluster-users at gluster.org Subject: RE: Files present on the backend but have become invisible from clients hi James, bricks 3-10 dont have problems, I think brick 01, 02 went to split brain situation, could you confirm if you see the following logs in your mount's log file [afr-self-heal-metadata.c:524:afr_sh_metadata_fix]0-stress-volume-replicate-0: Unable to self-heal permissions/ownership of '/' (possible split-brain). Please fix the file on all backend volumes. Pranith. ________________________________________ From: Burnash, James [jburnash at knight.com] Sent: Monday, June 13, 2011 11:56 PM To: Pranith Kumar. Karampuri; Jeff Darcy (jdarcy at redhat.com); gluster-users at gluster.org Subject: RE: Files present on the backend but have become invisible from clients Hi Pranith. Here is the revised listing - please notice that bricks g01 and g02 on the two servers (jc1letgfs14 and 15) have what appear to be "normal" trusted.afr attributes, but the balance of the bricks (3-10) all have =0x000000000000000000000000. http://pastebin.com/j0hVFTzd Is this right, or am I looking at this backwards / sideways? James Burnash Unix Engineer Knight Capital Group -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Burnash, James Sent: Monday, June 13, 2011 8:28 AM To: 'Pranith Kumar. Karampuri'; Jeff Darcy (jdarcy at redhat.com); gluster-users at gluster.org Subject: Re: Files present on the backend but have become invisible from clients Hi Pranith. Sorry - last week was a rough one. Disregard that pastebin - I will put up a new one that makes more sense and repost to the list. James -----Original Message----- From: Pranith Kumar. Karampuri [mailto:pranithk at gluster.com] Sent: Monday, June 13, 2011 1:12 AM To: Burnash, James; Jeff Darcy (jdarcy at redhat.com); gluster-users at gluster.org Subject: RE: Files present on the backend but have become invisible from clients hi James, I looked at the pastebin sample, I see that all of the attrs are complete zeros, Could you let me know what is it that I am missing. Pranith ________________________________________ From: gluster-users-bounces at gluster.org [gluster-users-bounces at gluster.org] on behalf of Burnash, James [jburnash at knight.com] Sent: Saturday, June 11, 2011 12:57 AM To: Jeff Darcy (jdarcy at redhat.com); gluster-users at gluster.org Subject: Re: Files present on the backend but have become invisible from clients Hi Jeff and Gluster users. Question about inconsistent looking attributes on brick directories on my Gluster backend servers. http://pastebin.com/b964zMu8 What stands out here is that the two original servers (jc1letgfs17 and 18) only show attributes of 0x000000000000000000000000 for every other pfs-ro1-client-X, while the two servers that were added some time ago show a different pattern (as can be seen in the pastebin sample). Configuration at end of this message: Any hints or suggestions would be greatly appreciated. Thanks, James Config: All on Gluster 3.1.3 Servers: 4 CentOS 5.5 (ProLiant DL370 G6 servers, Intel Xeon 3200 MHz), Each with: Single P812 Smart Array Controller, Single MDS600 with 70 2TB SATA drives configured as RAID 50 48 MB RAM Clients: 185 CentOS 5.2 (mostly DL360 G6). /pfs2 is the mount point for a Duplicated-Replicate volume of 4 servers. Volume Name: pfs-ro1 Type: Distributed-Replicate Status: Started Number of Bricks: 20 x 2 = 40 Transport-type: tcp Bricks: Brick1: jc1letgfs17-pfs1:/export/read-only/g01 Brick2: jc1letgfs18-pfs1:/export/read-only/g01 Brick3: jc1letgfs17-pfs1:/export/read-only/g02 Brick4: jc1letgfs18-pfs1:/export/read-only/g02 Brick5: jc1letgfs17-pfs1:/export/read-only/g03 Brick6: jc1letgfs18-pfs1:/export/read-only/g03 Brick7: jc1letgfs17-pfs1:/export/read-only/g04 Brick8: jc1letgfs18-pfs1:/export/read-only/g04 Brick9: jc1letgfs17-pfs1:/export/read-only/g05 Brick10: jc1letgfs18-pfs1:/export/read-only/g05 Brick11: jc1letgfs17-pfs1:/export/read-only/g06 Brick12: jc1letgfs18-pfs1:/export/read-only/g06 Brick13: jc1letgfs17-pfs1:/export/read-only/g07 Brick14: jc1letgfs18-pfs1:/export/read-only/g07 Brick15: jc1letgfs17-pfs1:/export/read-only/g08 Brick16: jc1letgfs18-pfs1:/export/read-only/g08 Brick17: jc1letgfs17-pfs1:/export/read-only/g09 Brick18: jc1letgfs18-pfs1:/export/read-only/g09 Brick19: jc1letgfs17-pfs1:/export/read-only/g10 Brick20: jc1letgfs18-pfs1:/export/read-only/g10 Brick21: jc1letgfs14-pfs1:/export/read-only/g01 Brick22: jc1letgfs15-pfs1:/export/read-only/g01 Brick23: jc1letgfs14-pfs1:/export/read-only/g02 Brick24: jc1letgfs15-pfs1:/export/read-only/g02 Brick25: jc1letgfs14-pfs1:/export/read-only/g03 Brick26: jc1letgfs15-pfs1:/export/read-only/g03 Brick27: jc1letgfs14-pfs1:/export/read-only/g04 Brick28: jc1letgfs15-pfs1:/export/read-only/g04 Brick29: jc1letgfs14-pfs1:/export/read-only/g05 Brick30: jc1letgfs15-pfs1:/export/read-only/g05 Brick11: jc1letgfs17-pfs1:/export/read-only/g06 Brick12: jc1letgfs18-pfs1:/export/read-only/g06 Brick13: jc1letgfs17-pfs1:/export/read-only/g07 Brick14: jc1letgfs18-pfs1:/export/read-only/g07 Brick15: jc1letgfs17-pfs1:/export/read-only/g08 Brick16: jc1letgfs18-pfs1:/export/read-only/g08 Brick17: jc1letgfs17-pfs1:/export/read-only/g09 Brick18: jc1letgfs18-pfs1:/export/read-only/g09 Brick19: jc1letgfs17-pfs1:/export/read-only/g10 Brick20: jc1letgfs18-pfs1:/export/read-only/g10 Brick21: jc1letgfs14-pfs1:/export/read-only/g01 Brick22: jc1letgfs15-pfs1:/export/read-only/g01 Brick23: jc1letgfs14-pfs1:/export/read-only/g02 Brick24: jc1letgfs15-pfs1:/export/read-only/g02 Brick25: jc1letgfs14-pfs1:/export/read-only/g03 Brick26: jc1letgfs15-pfs1:/export/read-only/g03 Brick27: jc1letgfs14-pfs1:/export/read-only/g04 Brick28: jc1letgfs15-pfs1:/export/read-only/g04 Brick29: jc1letgfs14-pfs1:/export/read-only/g05 Brick30: jc1letgfs15-pfs1:/export/read-only/g05 Brick31: jc1letgfs14-pfs1:/export/read-only/g06 Brick32: jc1letgfs15-pfs1:/export/read-only/g06 Brick33: jc1letgfs14-pfs1:/export/read-only/g07 Brick34: jc1letgfs15-pfs1:/export/read-only/g07 Brick35: jc1letgfs14-pfs1:/export/read-only/g08 Brick36: jc1letgfs15-pfs1:/export/read-only/g08 Brick37: jc1letgfs14-pfs1:/export/read-only/g09 Brick38: jc1letgfs15-pfs1:/export/read-only/g09 Brick39: jc1letgfs14-pfs1:/export/read-only/g10 Brick40: jc1letgfs15-pfs1:/export/read-only/g10 Options Reconfigured: diagnostics.brick-log-level: ERROR cluster.metadata-change-log: on diagnostics.client-log-level: ERROR performance.stat-prefetch: on performance.cache-size: 2GB network.ping-timeout: 10 DISCLAIMER: This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list Gluster-users at gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
