So - I continue to try to chase down this problem to its ultimate conclusion. The directories are all still visible to the users, but scanning for attributes of 0sAAAAAAAAAAAAAAAA still yielded matches on the set of GlusterFS servers. http://pastebin.com/mxvFnFj4 I tried running this command, but as you can see it wasn't happy, even though the syntax was correct: root at jc1letgfs17:~# gluster volume rebalance pfs-ro1 fix-layout start Usage: volume rebalance <VOLNAME> [fix-layout|migrate-data] {start|stop|status} I suspect this is a bug because of the "-" in my volume name. I'll test and confirm and file when I get a chance. So I just did the standard rebalance command: gluster volume rebalance pfs-ro1 start and it trundled along for a while and then one time when checked it's status, it failed: date; gluster volume rebalance pfs-ro1 status Thu May 26 09:02:00 EDT 2011 rebalance failed I re-ran it FOUR times getting a little farther with each attempt, and it eventually completed and then started doing the actual file migration part of the rebalance: Thu May 26 12:22:25 EDT 2011 rebalance step 1: layout fix in progress: fixed layout 779 Thu May 26 12:23:25 EDT 2011 rebalance step 2: data migration in progress: rebalanced 71 files of size 136518704 (total files scanned 57702) Now scanning for attributes of 0sAAAAAAAAAAAAAAAA yields less results, but some are still present: http://pastebin.com/x4wYq8ic As a possible sanity check, I did this command on my Read-Write GlusterFS storage servers (2 boxes, Distributed-Replicate), and got no "bad" attributes: jc1ladmin1:~/projects/gluster loop_check ' getfattr -dm - /export/read-only/g*' jc1letgfs{13,16} | egrep "jc1letgfs|0sAAAAAAAAAAAAAAAA$|file:" | less getfattr: /export/read-only/g*: No such file or directory getfattr: /export/read-only/g*: No such file or directory jc1letgfs13 jc1letgfs16 One difference in these two Storage server groups - the Read-Only group of 4 servers have their backend file systems formatted as XFS, while the Read-Write group of 2 are formatted with EXT4. Suggestions, critiques, etc gratefully solicited. James Burnash Unix Engineer. -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Burnash, James Sent: Monday, May 23, 2011 11:17 AM To: 'Mohit Anchlia' Cc: gluster-users at gluster.org Subject: Re: Files present on the backend but have become invisible from clients So - following up on this in the hope it will help others. According to the help offered (thanks!), the problem is that the attributes on the directories that cannot be seen by the clients are incorrect - which is to say they are all 0sAAAAAAAAAAAAAAAA. Here is the pastebin URL: http://pastebin.com/yz5PWjKV In desperation, I did a: gluster volume pfs-ro1 stop and then gluster volume pfs-ro1 start ... and now the directories and their contents are visible to the clients. I'm in the process of scanning the attributes of those directories again - I'll post another list on pastebin when I get them. James Burnash -----Original Message----- From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Burnash, James Sent: Thursday, May 19, 2011 3:47 PM To: 'Mohit Anchlia' Cc: gluster-users at gluster.org Subject: Re: Files present on the backend but have become invisible from clients
