Hello All-
I am having problems with rebalance ... fix-layout in version 3.2.5. I
extended a volume with add-brick but the fix-layout stalls after a small
number of layout fixes and does not make any more progress. I have
tried the operation twice on different servers with the same result.
The following errors are found in the fuse mount log file on the
server carrying out the operation.
[2011-12-28 21:38:14.840013] I
[afr-common.c:1038:afr_launch_self_heal] 0-nemo2-replicate-4:
background data self-heal triggered. path:
/users/hzu/DATA/ERAINT/ORCA025/2010/snow_ERAINT_2010.nc
[2011-12-28 21:38:15.93079] E
[client3_1-fops.c:1498:client3_1_fxattrop_cbk] 0-nemo2-client-8:
remote operation failed: Operation not supported
[2011-12-28 21:38:15.93141] E
[client3_1-fops.c:1498:client3_1_fxattrop_cbk] 0-nemo2-client-9:
remote operation failed: Operation not supported
[2011-12-28 21:38:15.93385] I
[client3_1-fops.c:1187:client3_1_fstat_cbk] 0-nemo2-client-8: remote
operation failed: Operation not supported
[2011-12-28 21:38:15.93521] I
[client3_1-fops.c:1187:client3_1_fstat_cbk] 0-nemo2-client-9: remote
operation failed: Operation not supported
The file in the error message is a link, and it is not broken as seen
from the volume mount point or the bricks.
There are some worrying error messages in the brick log files for
nemo2-client-8 and nemo2-client-9. Here are some exerpts from the
nemo2-client-8 log, which is similar to the 0-nemo2-client-9 log.
[2011-12-28 21:23:05.827877] W [posix.c:3928:do_xattrop]
0-nemo2-posix: Extended attributes not supported by filesystem
[2011-12-28 21:23:05.827932] I
[server3_1-fops.c:1705:server_fxattrop_cbk] 0-nemo2-server: 8438:
FXATTROP 0 (-2111276040) ==> -1 (Operation not support
ed)
[2011-12-28 21:23:05.828848] E [posix.c:4200:posix_fstat]
0-nemo2-posix: fstat failed on fd=0x2aaaac703804: Operation not
supported
[2011-12-28 21:23:05.828879] I
[server3_1-fops.c:1113:server_fstat_cbk] 0-nemo2-server: 8439: FSTAT
0 (-2111276040) ==> -1 (Operation not supported)
[2011-12-28 21:29:29.871213] W
[socket.c:1494:__socket_proto_state_machine] 0-tcp.nemo2-server:
reading from socket failed. Error (Transport endpoint i
s not connected), peer (192.171.166.81:1003)
[2011-12-28 21:29:29.871305] I
[server-helpers.c:360:do_lock_table_cleanup] 0-nemo2-server: inodelk
released on /users/hzu/DATA/ERAINT/ORCA025/2010/sno
w_ERAINT_2010.nc
[2011-12-28 21:29:29.871345] I [server-helpers.c:485:do_fd_cleanup]
0-nemo2-server: fd cleanup on
/users/hzu/DATA/ERAINT/ORCA025/2010/snow_ERAINT_2010.
nc
[2011-12-28 21:34:36.190023] I [server-helpers.c:485:do_fd_cleanup]
0-nemo2-server: fd cleanup on /
[2011-12-28 21:34:36.190055] I [server-helpers.c:485:do_fd_cleanup]
0-nemo2-server: fd cleanup on /users
[2011-12-28 21:34:36.190086] I [server-helpers.c:485:do_fd_cleanup]
0-nemo2-server: fd cleanup on /users/hzu
[2011-12-28 21:34:36.190102] I [server-helpers.c:485:do_fd_cleanup]
0-nemo2-server: fd cleanup on /users/hzu/DATA
[2011-12-28 21:34:36.190135] I [server-helpers.c:485:do_fd_cleanup]
0-nemo2-server: fd cleanup on /users/hzu/DATA/ERAINT
[2011-12-28 21:34:36.190154] I [server-helpers.c:485:do_fd_cleanup]
0-nemo2-server: fd cleanup on /users/hzu/DATA/ERAINT/ORCA025
[2011-12-28 21:34:36.190171] I [server-helpers.c:485:do_fd_cleanup]
0-nemo2-server: fd cleanup on /users/hzu/DATA/ERAINT/ORCA025/2009
[2011-12-28 21:38:15.92433] I
[server3_1-fops.c:1705:server_fxattrop_cbk] 0-nemo2-server: 12228:
FXATTROP 7 (-2111276040) ==> -1 (Operation not supported)
[2011-12-28 21:38:15.92743] E [posix.c:4200:posix_fstat]
0-nemo2-posix: fstat failed on fd=0x2aaaac703804: Operation not
supported
[2011-12-28 21:38:15.92775] I
[server3_1-fops.c:1113:server_fstat_cbk] 0-nemo2-server: 12229:
FSTAT 7 (-2111276040) ==> -1 (Operation not supported)
The backend filesystems are ext4 and the are mounted with options
"acl,user_xattr". I tested extended attribute support (as suggested
here:
http://gluster.org/pipermail/gluster-users/2010-December/006257.html)
and could not find any problems, so I don't understand the "Extended
attributes not supported by filesystem" error. The only unusual thing
about the filesystems is the reduced number of filesystem features
enabled compared to other bricks. These are the ext4 features enabled.
has_journal ext_attr resize_inode dir_index filetype needs_recovery
sparse_super large_file
All the other bricks in the volume have these features plus extent,
flex_bg, huge_file, uninit_bg, dir_nlink and extra_isize. I don't know
if any of these missing ext4 features are part of the problem. Does
anybody know what's going on here?
Regards
Dan.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gluster.org/pipermail/gluster-users/attachments/20111229/a5cecfb0/attachment.htm>
