On Sunday, March 20, 2016 14:44:18 Atin Mukherjee wrote:
> What does gluster peer status output say?
>From the node that failed:
[root@vmserver-b ~]# gluster peer status
Number of Peers: 1
Hostname: 192.168.122.10
Uuid: d8e1d7a0-077a-4a50-93f6-d3922e3b96b9
State: Accepted peer request (Connected)
> Also send the glusterd log of the node where the commands have failed.
The two logs are attached. The peer status says connected. The log file says
"FAILED : Host 192.168.122.10 is not in 'Peer in Cluster' state" I'm
confused. :)
Thanks for your help on this!
j
--
Joshua J. Kugler - Fairbanks, Alaska
Azariah Enterprises - Programming and Website Design
joshua@xxxxxxxxxxx - Jabber: pedahzur@xxxxxxxxx
PGP Key: http://pgp.mit.edu/ ID 0x73B13B6A
[2016-03-22 05:26:40.008570] I [cli-cmd-volume.c:1804:cli_check_gsync_present] 0-: geo-replication not installed
[2016-03-22 05:26:40.008867] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:26:40.009394] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2016-03-22 05:26:40.016177] I [cli-cmd-volume.c:1804:cli_check_gsync_present] 0-: geo-replication not installed
[2016-03-22 05:26:40.016484] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:26:40.017430] I [cli-rpc-ops.c:588:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2016-03-22 05:26:40.017541] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2016-03-22 05:26:40.027743] I [cli-cmd-volume.c:1804:cli_check_gsync_present] 0-: geo-replication not installed
[2016-03-22 05:26:40.028861] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:26:43.027142] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:26:46.027459] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:26:49.027722] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:26:52.028026] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:26:55.028325] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:26:58.028535] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:01.028799] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:04.029077] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:07.029260] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:10.031382] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:13.031663] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:16.031968] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:19.032261] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:22.032594] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:25.032751] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:28.033138] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:31.033366] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:34.033645] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:37.033880] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:40.034188] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:43.034510] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:43.370555] I [cli-rpc-ops.c:131:gf_cli_probe_cbk] 0-cli: Received resp to probe
[2016-03-22 05:27:43.370659] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2016-03-22 05:27:43.922428] I [cli-cmd-volume.c:1804:cli_check_gsync_present] 0-: geo-replication not installed
[2016-03-22 05:27:43.922737] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:43.972665] I [cli-rpc-ops.c:131:gf_cli_probe_cbk] 0-cli: Received resp to probe
[2016-03-22 05:27:43.972731] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2016-03-22 05:27:43.985455] I [cli-cmd-volume.c:1804:cli_check_gsync_present] 0-: geo-replication not installed
[2016-03-22 05:27:43.985690] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:43.994157] I [cli-cmd-volume.c:402:cli_cmd_volume_create_cbk] 0-cli: Replicate cluster type found. Checking brick order.
[2016-03-22 05:27:43.994315] I [cli-cmd-volume.c:309:cli_cmd_check_brick_order] 0-cli: Brick order okay
[2016-03-22 05:27:44.002605] I [cli-rpc-ops.c:892:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
[2016-03-22 05:27:44.002843] I [input.c:36:cli_batch] 0-: Exiting with: -1
[2016-03-22 05:27:44.712793] I [cli-cmd-volume.c:1804:cli_check_gsync_present] 0-: geo-replication not installed
[2016-03-22 05:27:44.713003] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:44.715763] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2016-03-22 05:27:44.740790] I [cli-cmd-volume.c:1804:cli_check_gsync_present] 0-: geo-replication not installed
[2016-03-22 05:27:44.741046] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:44.745984] I [cli-rpc-ops.c:588:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0
[2016-03-22 05:27:44.746300] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2016-03-22 05:27:44.773014] I [cli-cmd-volume.c:1804:cli_check_gsync_present] 0-: geo-replication not installed
[2016-03-22 05:27:44.773221] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:44.777564] I [cli-rpc-ops.c:131:gf_cli_probe_cbk] 0-cli: Received resp to probe
[2016-03-22 05:27:44.777608] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2016-03-22 05:27:44.803990] I [cli-cmd-volume.c:1804:cli_check_gsync_present] 0-: geo-replication not installed
[2016-03-22 05:27:44.804227] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 05:27:44.808039] I [cli-cmd-volume.c:402:cli_cmd_volume_create_cbk] 0-cli: Replicate cluster type found. Checking brick order.
[2016-03-22 05:27:44.808124] I [cli-cmd-volume.c:309:cli_cmd_check_brick_order] 0-cli: Brick order okay
[2016-03-22 05:27:44.809374] I [cli-rpc-ops.c:892:gf_cli_create_volume_cbk] 0-cli: Received resp to create volume
[2016-03-22 05:27:44.809509] I [input.c:36:cli_batch] 0-: Exiting with: -1
[2016-03-22 07:41:51.879558] I [cli-cmd-volume.c:1804:cli_check_gsync_present] 0-: geo-replication not installed
[2016-03-22 07:41:51.880096] I [socket.c:2353:socket_event_handler] 0-transport: disconnecting now
[2016-03-22 07:41:51.909851] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2016-03-22 05:26:38.596312] I [MSGID: 100030] [glusterfsd.c:2035:main] 0-glusterd: Started running glusterd version 3.6.9 (args: glusterd --xlator-option *.upgrade=on -N)
[2016-03-22 05:26:38.598317] I [graph.c:269:gf_add_cmdline_options] 0-management: adding option 'upgrade' for volume 'management' with value 'on'
[2016-03-22 05:26:38.598364] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536
[2016-03-22 05:26:38.598384] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory
[2016-03-22 05:26:38.602322] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.9/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2016-03-22 05:26:38.602322] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2016-03-22 05:26:38.602322] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2016-03-22 05:26:38.602822] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system
[2016-03-22 05:26:38.603074] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
[2016-03-22 05:26:38.603089] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
[2016-03-22 05:26:38.603095] I [glusterd-store.c:2068:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 30603
[2016-03-22 05:26:38.603312] I [glusterd-store.c:3502:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list.
[2016-03-22 05:26:38.603312] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options, returned error: (No such file or directory)
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option transport.socket.listen-backlog 128
7: option upgrade on
8: option ping-timeout 30
9: option transport.socket.read-fail-log off
10: option transport.socket.keepalive-interval 2
11: option transport.socket.keepalive-time 10
12: option transport-type rdma
13: option working-directory /var/lib/glusterd
14: end-volume
15:
+------------------------------------------------------------------------------+
[2016-03-22 05:26:38.648169] W [glusterfsd.c:1211:cleanup_and_exit] (--> 0-: received signum (15), shutting down
[2016-03-22 05:26:39.526487] I [MSGID: 100030] [glusterfsd.c:2035:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.6.9 (args: /usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2016-03-22 05:26:39.528642] I [glusterd.c:1214:init] 0-management: Maximum allowed open file descriptors set to 65536
[2016-03-22 05:26:39.528668] I [glusterd.c:1259:init] 0-management: Using /var/lib/glusterd as working directory
[2016-03-22 05:26:39.531746] E [rpc-transport.c:266:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.6.9/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2016-03-22 05:26:39.531760] W [rpc-transport.c:270:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2016-03-22 05:26:39.531766] W [rpcsvc.c:1524:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2016-03-22 05:26:39.532349] I [glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system
[2016-03-22 05:26:39.532597] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
[2016-03-22 05:26:39.532616] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
[2016-03-22 05:26:39.532622] I [glusterd-store.c:2068:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 30603
[2016-03-22 05:26:39.532678] I [glusterd-store.c:3502:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list.
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option transport.socket.listen-backlog 128
7: option ping-timeout 30
8: option transport.socket.read-fail-log off
9: option transport.socket.keepalive-interval 2
10: option transport.socket.keepalive-time 10
11: option transport-type rdma
12: option working-directory /var/lib/glusterd
13: end-volume
14:
+------------------------------------------------------------------------------+
[2016-03-22 05:26:40.009185] I [glusterd-handler.c:1241:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2016-03-22 05:26:40.017348] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2016-03-22 05:26:40.029581] I [glusterd-handler.c:1031:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 192.168.122.10 24007
[2016-03-22 05:26:40.030218] I [glusterd-handler.c:3198:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: 192.168.122.10 (24007)
[2016-03-22 05:26:40.079638] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2016-03-22 05:26:40.081873] I [glusterd-handler.c:3131:glusterd_friend_add] 0-management: connect returned 0
[2016-03-22 05:26:40.142688] E [store.c:432:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info, returned error: (No such file or directory)
[2016-03-22 05:26:40.142764] I [glusterd.c:176:glusterd_uuid_generate_save] 0-management: generated UUID: fe722085-ac0f-4449-a43f-2dc9dd1fd8fb
[2016-03-22 05:26:40.236825] I [glusterd-rpc-ops.c:245:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: d8e1d7a0-077a-4a50-93f6-d3922e3b96b9, host: 192.168.122.10
[2016-03-22 05:26:40.274183] I [glusterd-rpc-ops.c:387:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
[2016-03-22 05:27:43.358827] I [glusterd-rpc-ops.c:437:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: d8e1d7a0-077a-4a50-93f6-d3922e3b96b9, host: 192.168.122.10, port: 0
[2016-03-22 05:27:43.972325] I [glusterd-handler.c:1031:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 192.168.122.11 24007
[2016-03-22 05:27:44.002269] E [glusterd-utils.c:7995:glusterd_new_brick_validate] 0-management: Host 192.168.122.10 is not in 'Peer in Cluster' state
[2016-03-22 05:27:44.002309] E [glusterd-volume-ops.c:896:glusterd_op_stage_create_volume] 0-management: Host 192.168.122.10 is not in 'Peer in Cluster' state
[2016-03-22 05:27:44.002406] E [glusterd-syncop.c:1184:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host 192.168.122.10 is not in 'Peer in Cluster' state
[2016-03-22 05:27:44.715206] I [glusterd-handler.c:1241:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2016-03-22 05:27:44.745899] I [glusterd-handler.c:1296:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2016-03-22 05:27:44.777379] I [glusterd-handler.c:1031:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 192.168.122.11 24007
[2016-03-22 05:27:44.809237] E [glusterd-utils.c:7995:glusterd_new_brick_validate] 0-management: Host 192.168.122.10 is not in 'Peer in Cluster' state
[2016-03-22 05:27:44.809252] E [glusterd-volume-ops.c:896:glusterd_op_stage_create_volume] 0-management: Host 192.168.122.10 is not in 'Peer in Cluster' state
[2016-03-22 05:27:44.809258] E [glusterd-syncop.c:1184:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host 192.168.122.10 is not in 'Peer in Cluster' state
[2016-03-22 07:41:51.907592] I [glusterd-handler.c:1241:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
