Re: Expanding Volumes

Atin Mukherjee <atin.mukherjee83@xxxxxxxxx> · Tue, 9 Jun 2015 18:54:37 +0530

Rebalance is broken in 3.7.1. It will be fixed in 3.7.2. Sorry for the inconvenience.
Regards,

Atin
Sent from Samsung Galaxy S4
On 9 Jun 2015 18:34, "Jonhnny Weslley" <jw@xxxxxxxxxxxxxxxxxx> wrote:
Hi guys,
I trying to create a pool of 4 nodes using centos7 and gluster 3.7 in a vagrant-based environment for test. First, I create and start a replicated volume using only 2 nodes (replication 2). After, I mount the volume using fuse and copy some files. Everything works fine. 

Then, I try to expand the volume previously created using the command:

sudo gluster volume add-brick jged 10.10.50.73:/home/vagrant/brick 10.10.50.74:/home/vagrant/brick force

And works again:

sudo gluster volume info

Volume Name: jged
Type: Distributed-Replicate
Volume ID: 862ab9b7-4753-4682-ba44-cbe481b1b7df
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.10.50.71:/home/vagrant/brick
Brick2: 10.10.50.72:/home/vagrant/brick
Brick3: 10.10.50.73:/home/vagrant/brick
Brick4: 10.10.50.74:/home/vagrant/brick
Options Reconfigured:
performance.readdir-ahead: on

But when I try to rebalance the volume (sudo gluster volume rebalance jged start), the gluster's  proccess in the node where the command was executed dies and dont start again after running 'systemctl start glusterd'. I look the log file (/var/log/glusterfs/etc-glusterfs-glusterd.vol.log) but I cant figure out what is wrong! :(

Follow the tail of the log file:

[2015-06-09 12:30:56.197802] I [MSGID: 100030] [glusterfsd.c:2294:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid)
[2015-06-09 12:30:56.207596] I [glusterd.c:1282:init] 0-management: Maximum allowed open file descriptors set to 65536
[2015-06-09 12:30:56.207653] I [glusterd.c:1327:init] 0-management: Using /var/lib/glusterd as working directory
[2015-06-09 12:30:56.211505] E [rpc-transport.c:291:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.7.1/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2015-06-09 12:30:56.211521] W [rpc-transport.c:295:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2015-06-09 12:30:56.211528] W [rpcsvc.c:1595:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2015-06-09 12:30:56.211535] E [glusterd.c:1515:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2015-06-09 12:30:56.213311] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system
[2015-06-09 12:30:56.213454] I [glusterd-store.c:1986:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30700
[2015-06-09 12:30:56.213523] I [glusterd.c:154:glusterd_uuid_init] 0-management: retrieved UUID: f264d968-5a14-459b-8f3b-569aa15c3ce2
[2015-06-09 12:30:56.213568] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2015-06-09 12:30:56.213675] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2015-06-09 12:30:56.213801] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2015-06-09 12:30:56.213896] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2015-06-09 12:30:56.213979] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2015-06-09 12:30:56.214094] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2015-06-09 12:30:56.987649] I [glusterd-handler.c:3387:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2015-06-09 12:30:56.987711] I [glusterd-handler.c:3387:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2015-06-09 12:30:56.987755] I [glusterd-handler.c:3387:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2015-06-09 12:30:56.987801] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-06-09 12:30:56.989874] W [socket.c:923:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 13, Invalid argument
[2015-06-09 12:30:56.989890] E [socket.c:3015:socket_connect] 0-management: Failed to set keep-alive: Invalid argument
[2015-06-09 12:30:56.990051] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-06-09 12:30:56.992360] W [socket.c:923:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 14, Invalid argument
[2015-06-09 12:30:56.992419] E [socket.c:3015:socket_connect] 0-management: Failed to set keep-alive: Invalid argument
[2015-06-09 12:30:56.992629] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-06-09 12:30:56.994163] W [socket.c:923:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument
[2015-06-09 12:30:56.994177] E [socket.c:3015:socket_connect] 0-management: Failed to set keep-alive: Invalid argument
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option transport.socket.listen-backlog 128
  7:     option ping-timeout 30
  8:     option transport.socket.read-fail-log off
  9:     option transport.socket.keepalive-interval 2
 10:     option transport.socket.keepalive-time 10
 11:     option transport-type rdma
 12:     option working-directory /var/lib/glusterd
 13: end-volume
 14:
+------------------------------------------------------------------------------+
[2015-06-09 12:30:57.007380] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-06-09 12:30:57.288783] I [glusterd-rpc-ops.c:464:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 41707504-c217-4b7a-8f3a-7150bd040d0a, host: 10.10.50.74, port: 0
[2015-06-09 12:30:57.292480] I [glusterd-handler.c:2587:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 41707504-c217-4b7a-8f3a-7150bd040d0a
[2015-06-09 12:30:57.292542] I [glusterd-handler.c:2630:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2015-06-09 12:30:57.293810] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2015-06-09 12:30:57.303585] W [socket.c:3059:socket_connect] 0-nfs: Ignore failed connection attempt on , (No such file or directory)
[2015-06-09 12:30:57.304172] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2015-06-09 12:30:57.324624] W [socket.c:3059:socket_connect] 0-glustershd: Ignore failed connection attempt on , (No such file or directory)
[2015-06-09 12:30:57.324721] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2015-06-09 12:30:57.324767] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2015-06-09 12:30:57.343225] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash:
2015-06-09 12:30:57
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7.1
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb2)[0x7f3f46ca2d92]
/lib64/libglusterfs.so.0(gf_print_trace+0x32d)[0x7f3f46cbd9ed]
/lib64/libc.so.6(+0x35a00)[0x7f3f45696a00]
/lib64/libc.so.6(gsignal+0x39)[0x7f3f45696989]
/lib64/libc.so.6(abort+0x148)[0x7f3f45698098]
/lib64/libc.so.6(+0x76197)[0x7f3f456d7197]
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f3f4576ecb7]
/lib64/libc.so.6(+0x10be80)[0x7f3f4576ce80]
/lib64/libc.so.6(+0x10b5bb)[0x7f3f4576c5bb]
/lib64/libc.so.6(__snprintf_chk+0x78)[0x7f3f4576c4d8]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_volume_defrag_restart+0x191)[0x7f3f3bb37931]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_restart_rebalance+0x82)[0x7f3f3bb3daa2]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_spawn_daemons+0x4f)[0x7f3f3bb3db1f]
/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7f3f46cdd482]
/lib64/libc.so.6(+0x47570)[0x7f3f456a8570]
---------

-- 
Jonhnny Weslley
http://blog.jonhnnyweslley.net

_______________________________________________

Gluster-devel mailing list

Gluster-devel@xxxxxxxxxxx

http://www.gluster.org/mailman/listinfo/gluster-devel

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-devel