# rpm -qa | grep -i gluster glusterfs-libs-3.6.2-1.el6.x86_64 glusterfs-api-3.6.2-1.el6.x86_64 glusterfs-fuse-3.6.2-1.el6.x86_64 glusterfs-server-3.6.2-1.el6.x86_64 glusterfs-cli-3.6.2-1.el6.x86_64 glusterfs-geo-replication-3.6.2-1.el6.x86_64 glusterfs-3.6.2-1.el6.x86_64 /var/log/glusterfs/cli.log has some interesting data in it: [2015-02-24 15:01:41.902048] D [cli.c:612:cli_rpc_init] 0-cli: Connecting to glusterd using default socket [2015-02-24 15:01:41.902163] D [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2015-02-24 15:01:41.902190] D [rpc-clnt.c:986:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2015-02-24 15:01:41.902221] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so [2015-02-24 15:01:41.904940] D [socket.c:3684:socket_init] 0-glusterfs: disabling nodelay [2015-02-24 15:01:41.904973] D [socket.c:3799:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2015-02-24 15:01:41.904999] D [socket.c:3802:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2015-02-24 15:01:41.905014] D [socket.c:3819:socket_init] 0-glusterfs: using system polling thread [2015-02-24 15:01:41.905044] T [rpc-clnt.c:418:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2015-02-24 15:01:41.905066] T [socket.c:2871:socket_connect] 0-glusterfs: connecting 0x1f33f30, state=0 gen=0 sock=-1 [2015-02-24 15:01:41.905090] T [name.c:290:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /var/run/glusterd.socket [2015-02-24 15:01:41.905117] T [name.c:106:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2015-02-24 15:01:41.905278] D [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glusterfs: defaulting frame-timeout to 30mins [2015-02-24 15:01:41.905302] D [rpc-clnt.c:986:rpc_clnt_connection_init] 0-glusterfs: disable ping-timeout [2015-02-24 15:01:41.905326] D [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so [2015-02-24 15:01:41.905368] D [socket.c:3799:socket_init] 0-glusterfs: SSL support on the I/O path is NOT enabled [2015-02-24 15:01:41.905387] D [socket.c:3802:socket_init] 0-glusterfs: SSL support for glusterd is NOT enabled [2015-02-24 15:01:41.905408] D [socket.c:3819:socket_init] 0-glusterfs: using system polling thread [2015-02-24 15:01:41.905429] T [rpc-clnt.c:418:rpc_clnt_reconnect] 0-glusterfs: attempting reconnect [2015-02-24 15:01:41.905448] T [socket.c:2871:socket_connect] 0-glusterfs: connecting 0x1f3c3e0, state=0 gen=0 sock=-1 [2015-02-24 15:01:41.905469] T [name.c:290:af_unix_client_get_remote_sockaddr] 0-glusterfs: using connect-path /tmp/quotad.socket [2015-02-24 15:01:41.905501] T [name.c:106:af_unix_client_bind] 0-glusterfs: bind-path not specified for unix socket, letting connect to assign default value [2015-02-24 15:01:41.905551] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905592] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905614] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905635] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905656] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905691] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905717] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905741] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905767] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905789] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905810] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905849] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905866] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905883] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:41.905900] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067441] D [cli-cmd-volume.c:1787:cli_check_gsync_present] 0-cli: Returning 0 [2015-02-24 15:01:42.067547] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067576] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067617] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067653] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067680] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067707] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067733] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067752] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067775] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067797] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067821] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067843] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067878] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067898] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067914] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067938] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067962] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.067983] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068002] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068025] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068040] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068055] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068070] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068083] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068099] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068114] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068131] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068153] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068168] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068191] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068216] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068236] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068254] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068275] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068297] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068328] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068350] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068372] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.068395] D [registry.c:408:cli_cmd_register] 0-cli: Returning 0 [2015-02-24 15:01:42.069226] T [cli.c:264:cli_rpc_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2015-02-24 15:01:42.069271] T [cli-quotad-client.c:94:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_CONNECT [2015-02-24 15:01:42.069300] I [socket.c:2344:socket_event_handler] 0-transport: disconnecting now [2015-02-24 15:01:42.069338] T [cli-quotad-client.c:100:cli_quotad_notify] 0-glusterfs: got RPC_CLNT_DISCONNECT [2015-02-24 15:01:42.069455] T [rpc-clnt.c:1381:rpc_clnt_record] 0-glusterfs: Auth Info: pid: 0, uid: 0, gid: 0, owner: [2015-02-24 15:01:42.069499] T [rpc-clnt.c:1238:rpc_clnt_record_build_header] 0-rpc-clnt: Request fraglen 220, payload: 156, rpc hdr: 64 [2015-02-24 15:01:42.069915] T [socket.c:2863:socket_connect] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f9d1ceb3550] (--> /usr/lib64/glusterfs/3.6.2/rpc-transport/socket.so(+0x7293)[0x7f9d1a142293] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x468)[0x7f9d1c60bfe8] (--> gluster(cli_submit_request+0xdb)[0x40a96b] (--> gluster(cli_cmd_submit+0x8e)[0x40b76e] ))))) 0-glusterfs: connect () called on transport already connected [2015-02-24 15:01:42.070711] T [rpc-clnt.c:1573:rpc_clnt_submit] 0-rpc-clnt: submitted request (XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 11) to rpc-transport (glusterfs) [2015-02-24 15:01:42.070739] D [rpc-clnt-ping.c:231:rpc_clnt_start_ping] 0-glusterfs: ping timeout is 0, returning [2015-02-24 15:01:42.078829] T [rpc-clnt.c:660:rpc_clnt_reply_init] 0-glusterfs: received rpc message (RPC XID: 0x1 Program: Gluster CLI, ProgVers: 2, Proc: 11) from rpc-transport (glusterfs) [2015-02-24 15:01:42.079096] T [cli-rpc-ops.c:1358:gf_cli_print_rebalance_status] 0-cli: failed to get status [2015-02-24 15:01:42.079118] E [cli-rpc-ops.c:1973:gf_cli3_remove_brick_status_cbk] 0-cli: Failed to print remove-brick rebalance status [2015-02-24 15:01:42.079213] D [cli-cmd.c:384:cli_cmd_submit] 0-cli: Returning -2 [2015-02-24 15:01:42.079242] D [cli-rpc-ops.c:3592:gf_cli_remove_brick] 0-cli: Returning -2 [2015-02-24 15:01:42.079260] D [cli-cmd-volume.c:1557:cli_cmd_volume_remove_brick_cbk] 0-cli: frame->local is not NULL (0x7f9d0c0009c0) [2015-02-24 15:01:42.079287] I [input.c:36:cli_batch] 0-: Exiting with: -2 /var/log/glusterfs/etc-glusterfs-glusterd.vol.log is has repeats of: 2015-02-24 14:53:23.852588] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) [2015-02-24 14:53:26.894893] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) The message "I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer a32a46c8-d2d4-4b5d-8f7b-ee6dee3a15e0, in Peer in Cluster state, has disconnected from glusterd." repeated 34 times between [2015-02-24 14:51:32.621065] and [2015-02-24 14:53:22.852790] The message "I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd." repeated 39 times between [2015-02-24 14:51:27.420758] and [2015-02-24 14:53:26.894928] [2015-02-24 14:53:28.895747] I [MSGID: 106004] [glusterd-handler.c:4365:__glusterd_peer_rpc_notify] 0-management: Peer a32a46c8-d2d4-4b5d-8f7b-ee6dee3a15e0, in Peer in Cluster state, has disconnected from glusterd. [2015-02-24 14:53:29.895225] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) [2015-02-24 14:53:29.895264] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd. [2015-02-24 14:53:32.922976] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) [2015-02-24 14:53:35.952776] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) [2015-02-24 14:53:38.983371] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) [2015-02-24 14:53:42.077272] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) [2015-02-24 14:53:45.129492] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) [2015-02-24 14:53:48.327998] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) [2015-02-24 14:53:51.368593] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) [2015-02-24 14:53:54.521800] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) [2015-02-24 14:53:57.560446] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) [2015-02-24 14:54:00.643818] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) [2015-02-24 14:54:03.675623] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) [2015-02-24 14:54:06.708956] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/b115a9ba4d9b29282b5c7d7908751b35.socket failed (Invalid argument) On Tue, Feb 24, 2015 at 1:18 AM, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote: > Could you attach glusterd logs? Which version of glusterfs-server are > you using? > > ~Atin > > On 02/24/2015 12:59 PM, Sam Giraffe wrote: >> I get the same blank message when I try brick status with both bricks: >> >> # gluster volume remove-brick art server1:/brick1 server2:/brick2 status >> >> Node Rebalanced-files size scanned failures >> skipped status run time in secs >> --------- ----------- ----------- ----------- ----------- >> ----------- ------------ -------------- >> >> On Mon, Feb 23, 2015 at 8:17 PM, Atin Mukherjee <amukherj@xxxxxxxxxx> wrote: >>> >>> >>> On 02/24/2015 06:41 AM, Sam Giraffe wrote: >>>> Hi, >>>> >>>> On my 20 nodes with 2 replica cluster I was able to run: >>>> >>>> # gluster volume remove-brick art server1:/brick1 server2:/brick2 start >>>> >>>> I got a message stating: >>>> >>>> volume remove-brick start: success >>>> ID: 16b887bb-e848-4054-a5af-9390055d32c9 >>>> >>>> However, when I run 'status' I get nothing in the output: >>>> >>>> $ gluster volume remove-brick art server1:/brick1 status >>>> Node Rebalanced-files size scanned failures >>>> skipped status run time in secs >>>> --------- ----------- >>>> ----------- ----------- ----------- ----------- >>>> ------------ -------------- >>> In remove-brick start you have couple of bricks to be removed however >>> the status command has only one, Could you try remove brick status with >>> correct brick details i.e. appending brick2 in the status query. >>>> >>>> And when I try a commit I get an error saying that the removal is in progress: >>>> >>>> # gluster volume remove-brick art server1:/brick1 server2:/brick2 commit >>>> Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y >>>> volume remove-brick commit: failed: Staging failed on server1. Error: >>>> use 'force' option as migration is in progress >>>> Staging failed on server2. Error: use 'force' option as migration is in progress >>>> >>> This is expected as remove-brick transaction hasn't finished. >>>> Any clue on how to get status of brick removal? >>>> >>>> Thank you >>>> _______________________________________________ >>>> Gluster-users mailing list >>>> Gluster-users@xxxxxxxxxxx >>>> http://www.gluster.org/mailman/listinfo/gluster-users >>>> >>> >>> -- >>> ~Atin >> >> > > -- > ~Atin _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
