OK, thanks for the info!
Regards,
Olav
On 11/06/14 08:38, Pranith Kumar Karampuri wrote:
On 06/11/2014 12:03 PM, Olav Peeters wrote:
Thanks Pranith!
I see this at the end of the log files of one of the problem bricks
(the first two errors are repeated several times):
[2014-06-10 09:55:28.354659] E [rpcsvc.c:1206:rpcsvc_submit_generic]
0-rpc-service: failed to submit message (XID: 0x103c59, Program:
GlusterFS 3.3, ProgVers: 330, Proc: 30) to rpc-transport
(tcp.sr_vol01-server)
[2014-06-10 09:55:28.354683] E [server.c:190:server_submit_reply]
(-->/usr/lib64/glusterfs/3.5.0/xlator/performance/io-threads.so(iot_finodelk_cbk+0xb9)
[0x7f8c8e82f189]
(-->/usr/lib64/glusterfs/3.5.0/xlator/debug/io-stats.so(io_stats_finodelk_cbk+0xed)
[0x7f8c8e1f22ed]
(-->/usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_finodelk_cbk+0xad)
[0x7f8c8dfc555d]))) 0-: Reply submission failed
pending frames:
frame : type(0) op(30)
frame : type(0) op(30)
frame : type(0) op(30)
frame : type(0) op(30)
...
...
frame : type(0) op(30)
frame : type(0) op(30)
frame : type(0) op(30)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2014-06-10 09:55:28configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.5.0
/lib64/libc.so.6(+0x329a0)[0x7f8c94aac9a0]
/usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(grant_blocked_inode_locks+0xc1)[0x7f8c8ea54061]
/usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(pl_inodelk_client_cleanup+0x249)[0x7f8c8ea54569]
/usr/lib64/glusterfs/3.5.0/xlator/features/locks.so(+0x6f0a)[0x7f8c8ea49f0a]
/usr/lib64/libglusterfs.so.0(gf_client_disconnect+0x5d)[0x7f8c964d701d]
/usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_connection_cleanup+0x458)[0x7f8c8dfbda48]
/usr/lib64/glusterfs/3.5.0/xlator/protocol/server.so(server_rpc_notify+0x183)[0x7f8c8dfb9713]
/usr/lib64/libgfrpc.so.0(rpcsvc_handle_disconnect+0x105)[0x7f8c96261d35]
/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x1a0)[0x7f8c96263880]
/usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f8c96264f98]
/usr/lib64/glusterfs/3.5.0/rpc-transport/socket.so(+0xa9a1)[0x7f8c914c39a1]
/usr/lib64/libglusterfs.so.0(+0x672f7)[0x7f8c964d92f7]
/usr/sbin/glusterfsd(main+0x564)[0x4075e4]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f8c94a98d1d]
/usr/sbin/glusterfsd[0x404679]
---------
Again no info to be found online about the error.
Any idea?
This is because of bug 1089470 which is fixed for 3.5.1. Which will be
releasing shortly.
Pranith
Olav
On 11/06/14 04:42, Pranith Kumar Karampuri wrote:
Olav,
Check logs of the bricks to see why the bricks went down.
Pranith
On 06/11/2014 04:02 AM, Olav Peeters wrote:
Hi,
I upgraded from glusterfs 3.4 to 3.5 about 8 days ago. Everything
was running fine until this morning. In a fuse mount we were having
write issues. Creating and deleting files became an issue all of a
sudden without any new changes to the cluster.
In /var/log/glusterfs/glustershd.log every couple of seconds I'm
getting this:
[2014-06-10 22:23:52.055128] I [rpc-clnt.c:1685:rpc_clnt_reconfig]
0-sr_vol01-client-13: changing port to 49156 (from 0)
[2014-06-10 22:23:52.060153] E
[socket.c:2161:socket_connect_finish] 0-sr_vol01-client-13:
connection to ip-of-one-of-the-gluster-nodes:49156 failed
(Connection refused)
# gluster volume status sr_vol01
shows that two bricks of the 18 are offline.
rebalance fails
Iptables was stopped on all nodes
If I cd into the two bricks which are offline according to the
gluster v status, I can read/write without any problems... The
disks are clearly fine. They are mounted, they are available.
I cannot find much info online about the error.
Does anyone have an idea what could be wrong?
How can I get the two bricks back online?
Cheers,
Olav
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://supercolony.gluster.org/mailman/listinfo/gluster-users
