Files present on the backend but have become invisible from clients

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Not to be obnoxious, but nobody has any feedback on this issue?

-----Original Message-----
From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Burnash, James
Sent: Thursday, May 26, 2011 12:56 PM
To: 'Mohit Anchlia'
Cc: gluster-users at gluster.org
Subject: Re: Files present on the backend but have become invisible from clients

So - I continue to try to chase down this problem to its ultimate conclusion.

The directories are all still visible to the users, but scanning for attributes of 0sAAAAAAAAAAAAAAAA still yielded matches on the set of GlusterFS servers.

http://pastebin.com/mxvFnFj4

I tried running this command, but as you can see it wasn't happy, even though the syntax was correct:

root at jc1letgfs17:~# gluster volume rebalance pfs-ro1 fix-layout start
Usage: volume rebalance <VOLNAME> [fix-layout|migrate-data] {start|stop|status}

I suspect this is a bug because of the "-" in my volume name. I'll test and confirm and file when I get a chance.

So I just did the standard rebalance command:
 gluster volume rebalance pfs-ro1 start

and it trundled along for a while and then one time when checked it's status, it failed:
 date; gluster volume rebalance pfs-ro1 status  Thu May 26 09:02:00 EDT 2011  rebalance failed

I re-ran it FOUR times getting a little farther with each attempt, and it eventually completed and then started doing the actual file migration part of the rebalance:
 Thu May 26 12:22:25 EDT 2011
 rebalance step 1: layout fix in progress: fixed layout 779  Thu May 26 12:23:25 EDT 2011  rebalance step 2: data migration in progress: rebalanced 71 files of size 136518704 (total files scanned 57702)

Now scanning for attributes of 0sAAAAAAAAAAAAAAAA yields less results, but some are still present:

http://pastebin.com/x4wYq8ic

As a possible sanity check, I did this command on my Read-Write GlusterFS storage servers (2 boxes, Distributed-Replicate), and got no "bad" attributes:
 jc1ladmin1:~/projects/gluster  loop_check ' getfattr -dm - /export/read-only/g*' jc1letgfs{13,16} | egrep "jc1letgfs|0sAAAAAAAAAAAAAAAA$|file:" | less
 getfattr: /export/read-only/g*: No such file or directory
 getfattr: /export/read-only/g*: No such file or directory
 jc1letgfs13
 jc1letgfs16

One difference in these two Storage server groups - the Read-Only group of 4 servers have their backend file systems formatted as XFS, while the Read-Write group of 2 are formatted with EXT4.

Suggestions, critiques, etc gratefully solicited.

James Burnash
Unix Engineer.

-----Original Message-----
From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Burnash, James
Sent: Monday, May 23, 2011 11:17 AM
To: 'Mohit Anchlia'
Cc: gluster-users at gluster.org
Subject: Re: Files present on the backend but have become invisible from clients

So - following up on this in the hope it will help others.

According to the help offered (thanks!), the problem is that the attributes on the directories that cannot be seen by the clients are incorrect - which is to say they are all 0sAAAAAAAAAAAAAAAA.

Here is the pastebin URL: http://pastebin.com/yz5PWjKV

In desperation, I did a:

gluster volume pfs-ro1 stop

and then

gluster volume pfs-ro1 start

... and now the directories and their contents are visible to the clients.

I'm in the process of scanning the attributes of those directories again - I'll post another list on pastebin when I get them.

James Burnash

-----Original Message-----
From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Burnash, James
Sent: Thursday, May 19, 2011 3:47 PM
To: 'Mohit Anchlia'
Cc: gluster-users at gluster.org
Subject: Re: Files present on the backend but have become invisible from clients



[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux