Quick update:
rebooting gluster-0-1 did not solve the problem. The one
change I did notice is that now the only place I can write
is to one of the test directories I had previously created
which had not appeared on gluster-0-1. Now, trying to create
a new directory elsewhere results in the "No space left on device"
error.
Pat
Hi,
I am writing to see if there are any suggestions
as to what I should look at next to debug my
problem or what I should try to correct it.\
Would rebooting the gluster-0-1 brick help?
A short recap:
In early January the gluster-0-1 brick filled up
(the gluster-data brick was close to being full
but the gluster-0-0 had 2T space left).
I deleted files I knew to be on gluster-0-1 and
gluster-data (working through a client so as to
"go through" gluster).
Now the gluster-0-1 brick is reporting to gluster
that it is full when it clearly is not.
I have found that if I create a new directory,
I can write to it, but it only appears on the
other 2 bricks (gluster-data and gluster-0-0).
I appreciate any help in restoring the normal gluster
function. Thanks.
Pat
Hi,
I now have a change in the behavior of the gluster issue.
This afternoon I deleted a file that I knew was located
on the gluster-0-1 disk (the deleting command was issued
from a client). After that I was able to make a directoy
(empty) that appeared on all 3 bricks. My next step was
to copy a directory with files to the /gdata directory.
That appeared to be successful, but I looked and nothing
was written to the gluster-0-1 brick, not even the
new directory. That seems to be the new state, the client
still sees files on gluster-0-1 but when it writes to the
/gdata area it only writes to the gluster-data and gluster-0-0
bricks.
From the bricks log on gluster-0-1 I have the following lines
from when I successfully created the test directory
[2014-01-30 14:26:55.176965] W [posix-handle.c:538:posix_handle_soft]
0-gdata-posix: symlink
../../c9/14/c9144c1f-01a2-41ba-80cc-deec9aac03c5/test ->
/mseas-data-0-1/.glusterfs/d9/b8/d9b84fcb-a452-4f4e-bdd9-ba1a392a3844
failed (No space left on device)
[2014-01-30 14:26:55.205253] E [posix.c:969:posix_mkdir]
0-gdata-posix: setting gfid on /mseas-data-0-1/projects/atl/test failed
Then when I tried copying a full directory
[2014-01-30 14:30:35.716046] E [posix.c:962:posix_mkdir]
0-gdata-posix: mkdir of
/mseas-data-0-1/projects/atl/FV_Matlab_Framework_JingPJH/trunk/Save/mat/Bottom_Gravity_Current_28
failed: No space left on device
[2014-01-30 14:30:35.716133] I [server3_1-fops.c:529:server_mkdir_cbk]
0-gdata-server: 19027: MKDIR
/projects/atl/FV_Matlab_Framework_JingPJH/trunk/Save/mat/Bottom_Gravity_Current_28
(00000000-0000-0000-0000-000000000000) ==> -1 (No space left on device)
[2014-01-30 14:30:37.105372] I
[server3_1-fops.c:1747:server_setattr_cbk] 0-gdata-server: 19029:
SETATTR (null) (--) ==> -1 (No such file or directory)
That last line is repeated many times, followed by
[2014-01-30 14:31:57.587539] I
[server3_1-fops.c:1747:server_setattr_cbk] 0-gdata-server: 22077:
SETATTR (null) (--) ==> -1 (No such file or directory)
Does this suggest anything? Is there additional information
I should post?
Thanks,
Pat
Hi Joe,
Sorry to take so long in responding, but we had another emergency
that took all my time...
The subsampled brick log file from gluster-0-1 is available at
http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-1/bricks/mseas-data-0-1.log.1
The df results on gluster-0-1 are
[root@nas-0-1 glusterfs]# df /mseas-data-0-1/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb1 21484242304 20662833060 821409244 97%
/mseas-data-0-1
[root@nas-0-1 glusterfs]# df -i /mseas-data-0-1/
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sdb1 3290028528 4391552 3285636976 1% /mseas-data-0-1
I've also included the same time period log files from gluster-0-1 and
the other bricks in
http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-data/
http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-0/
http://mseas.mit.edu/download/phaley/GlusterUsers/gluster-0-1/
Thanks for the help!
Pat
1st, I don't think we need log data from November. Let's try to
share logs that are limited to an example of the error you're trying
to get help with. A 40Mb client log is unnecessary when it's only
the last 5k that's relevant.
2nd, your "server" logs are mostly more client logs, with the
exception of etc-glusterfs-glusterd.vol.log. The brick logs that
might actually show what's happening are absent.
(/var/log/glusterfs/bricks/*)
The last line of the client log does show that
gluster-0-1:/mseas-data-0-1 is reporting that it's out of space. On
gluster-0-1 please share the log
/var/log/glusterfs/bricks/gluster-0-1 but limit it to no more than
an hour surrounding 2014-01-21 11:50:16. Also on that server, please
share the result of "df /gluster-0-1 ; df -i /gluster-0-1".
On 01/21/2014 10:58 PM, Pat Haley wrote:
Hi Vijay,
I've put the log files in
http://mseas.mit.edu/download/phaley/GlusterUsers/server_logs/
http://mseas.mit.edu/download/phaley/GlusterUsers/client_logs/
On 01/21/2014 10:24 PM, Pat Haley wrote:
Hi Joe,
The peer status on all 3 showed the
proper connections. Doing the killall
and restart on all three bricks fixed
the N in the Online column. I then
did have to remount the gluster
filesystem on the client
Unfortunately my original problem remains.
I'm still getting "no space on left on device"
when I try to write even a small file to
the gluster filesystem.
What should I look at next?
Would it be possible to share the brick and client log files?
Thanks,
Vijay
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: phaley@xxxxxxx
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
