Re: 3 node 2 brick server keeps going offline. Runs excruciatingly slow when both bricks online

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Update 

The speed is back down, but when the gfs is mounted via nfs instead of through the gluster client, i get good speeds


Thank You Kindly,
Kaamesh
Bioinformatician
Novocraft Technologies Sdn Bhd
C-23A-05, 3 Two Square, Section 19, 46300 Petaling Jaya
Selangor Darul Ehsan
Malaysia
Mobile: +60176562635

On Thu, Feb 26, 2015 at 8:58 AM, Kaamesh Kamalaaharan <kaamesh@xxxxxxxxxxxxx> wrote:
Hi guys,

My glusterfs setup is across 3 servers (gfs1 ,gfs2, hpc1) with 2 bricks (on gfs1 and gfs2).

The reason i have 3 servers is that i wanted the gluster to stay online when one server goes down as i frequently lose my gfs2 server. I have no idea what causes this and im at a dead end.

I reinstalled glusterfs-server on my gfs2 server and i found that my performance dropped to the point my R/W speed is around 1.5 kB/S on my 10G LAN . Pressing tab when i ls into a folder takes around 20-30 seconds to autocomplete, sometimes longer. Previously i had blazing fast speeds. When i take gluster2 off the network, the speed is back up.

Both my servers are now on the fritz and the bricks keep going offline and i have no idea what to do. Can anyone help shed some light on this? I have no idea what i need to provide so ill provide the logs and config. 


::EDIT::
As i was typing, gluster went down again and when i brought it back up, the speed went back to normal. This didnt happen the last 3 or four times i restarted the servers.. I would still like to know what is going on...
::::::::



 gluster volume info

 Volume Name: gfsvolume
Type: Replicate
Volume ID: a29bd2fb-b1ef-4481-be10-c2f4faf4059b
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gfs1:/export/sda/brick
Brick2: gfs2:/export/sda/brick
Options Reconfigured:
performance.quick-read: off
network.ping-timeout: 30
network.frame-timeout: 90
performance.cache-max-file-size: 2MB
cluster.server-quorum-type: none
nfs.addr-namelookup: off
nfs.trusted-write: off
performance.write-behind-window-size: 4MB
cluster.data-self-heal-algorithm: diff
performance.cache-refresh-timeout: 60
performance.cache-size: 6442450944
cluster.quorum-type: fixed
auth.allow: 172.*
cluster.quorum-count: 1
cluster.server-quorum-ratio: 50%

/var/log//glusterfs/
 
[2015-02-26 00:13:36.207408] I [glusterd-volume-ops.c:481:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gfsvolume
[2015-02-26 00:13:36.211810] E [glusterd-syncop.c:101:gd_collate_errors] 0-: Commit failed on gfs2. Please check log file for details.
[2015-02-26 00:14:17.939391] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f13408f8e6d] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f1340fa6b50] (-->/us
r/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f1342832d55]))) 0-: received signum (15), shutting down
[2015-02-26 00:14:20.127606] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 (/usr/sbin/glusterd -p /var/run/glusterd.pid)
[2015-02-26 00:14:20.136366] I [glusterd.c:1148:init] 0-management: Using /var/lib/glusterd as working directory
[2015-02-26 00:14:20.137768] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled
[2015-02-26 00:14:20.137797] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread
[2015-02-26 00:14:20.139452] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
[2015-02-26 00:14:20.139483] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device
[2015-02-26 00:14:20.139497] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2015-02-26 00:14:20.139568] W [rpcsvc.c:1483:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2015-02-26 00:14:20.139719] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled
[2015-02-26 00:14:20.139741] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread
[2015-02-26 00:14:23.351872] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2
[2015-02-26 00:14:23.355502] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2015-02-26 00:14:23.355552] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2015-02-26 00:14:23.584714] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0
[2015-02-26 00:14:23.588106] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0
[2015-02-26 00:14:23.588223] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-02-26 00:14:23.588328] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled
[2015-02-26 00:14:23.588347] I [socket.c:3576:socket_init] 0-management: using system polling thread
[2015-02-26 00:14:23.589290] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-02-26 00:14:23.589378] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled
[2015-02-26 00:14:23.589397] I [socket.c:3576:socket_init] 0-management: using system polling thread
[2015-02-26 00:14:23.589885] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option transport.socket.listen-backlog 128
  7:     option transport.socket.read-fail-log off
  8:     option transport.socket.keepalive-interval 2
  9:     option transport.socket.keepalive-time 10
 10:     option transport-type rdma
 11:     option working-directory /var/lib/glusterd
 12: end-volume
 13: 
+------------------------------------------------------------------------------+
[2015-02-26 00:14:23.780644] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0
[2015-02-26 00:14:23.798102] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-02-26 00:14:23.798242] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled
[2015-02-26 00:14:23.798276] I [socket.c:3576:socket_init] 0-management: using system polling thread
[2015-02-26 00:14:24.800202] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error: Permission denied
[2015-02-26 00:14:24.800858] I [glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully
[2015-02-26 00:14:24.801261] I [glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully
[2015-02-26 00:14:24.801649] I [glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully
[2015-02-26 00:14:24.802028] I [glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully
[2015-02-26 00:14:24.802410] I [glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully
[2015-02-26 00:14:24.802793] I [glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully
[2015-02-26 00:14:24.806276] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-02-26 00:14:24.806387] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled
[2015-02-26 00:14:24.806406] I [socket.c:3576:socket_init] 0-management: using system polling thread
[2015-02-26 00:14:25.807341] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No such file or directory
[2015-02-26 00:14:25.810987] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-02-26 00:14:25.811100] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled[2015-02-26 00:14:25.811119] I [socket.c:3576:socket_init] 0-management: using system polling thread
[2015-02-26 00:14:25.811329] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c
[2015-02-26 00:14:25.811370] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1
[2015-02-26 00:14:25.811391] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
[2015-02-26 00:14:25.811407] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1
[2015-02-26 00:14:25.811510] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2
[2015-02-26 00:14:25.982101] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now
[2015-02-26 00:14:25.982172] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now
[2015-02-26 00:14:25.982265] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0
[2015-02-26 00:14:26.003332] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /export/sda/brick on port 49153
[2015-02-26 00:14:26.159051] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:26.159112] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1
[2015-02-26 00:14:26.159130] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
[2015-02-26 00:14:26.178634] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c
[2015-02-26 00:14:26.178794] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to gfs2 (0), ret: 0
[2015-02-26 00:14:26.194289] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1
[2015-02-26 00:14:26.194348] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2
[2015-02-26 00:14:26.208631] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c
[2015-02-26 00:14:26.373031] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:26.373101] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2
[2015-02-26 00:14:26.385510] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:26.385645] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to hpc1 (0), ret: 0
[2015-02-26 00:14:26.399963] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1
[2015-02-26 00:14:26.400010] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2
[2015-02-26 00:14:26.414448] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c
[2015-02-26 00:14:26.668753] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:32.167298] W [socket.c:522:__socket_rwv] 0-management: readv on 172.20.20.22:24007 failed (No data available)
[2015-02-26 00:14:33.590941] E [socket.c:2161:socket_connect_finish] 0-management: connection to 172.20.20.22:24007 failed (Connection refused)
[2015-02-26 00:14:39.060572] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f727130ce6d] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f72719bab50] (-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f7273246d55]))) 0-: received signum (15), shutting down
[2015-02-26 00:14:42.990079] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 (/usr/sbin/glusterd -p /var/run/glusterd.pid)
[2015-02-26 00:14:42.994671] I [glusterd.c:1148:init] 0-management: Using /var/lib/glusterd as working directory
[2015-02-26 00:14:42.996068] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled
[2015-02-26 00:14:42.996098] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread
[2015-02-26 00:14:42.996892] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
[2015-02-26 00:14:42.996919] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device
[2015-02-26 00:14:42.996932] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
 [2015-02-26 00:14:42.997004] W [rpcsvc.c:1483:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2015-02-26 00:14:42.997138] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled
[2015-02-26 00:14:42.997156] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread
[2015-02-26 00:14:46.319737] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2
[2015-02-26 00:14:46.323519] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2015-02-26 00:14:46.323567] E [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2015-02-26 00:14:46.561399] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0
[2015-02-26 00:14:46.564795] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0
[2015-02-26 00:14:46.564919] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-02-26 00:14:46.565035] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled
[2015-02-26 00:14:46.565054] I [socket.c:3576:socket_init] 0-management: using system polling thread
[2015-02-26 00:14:46.566216] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-02-26 00:14:46.566329] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled
[2015-02-26 00:14:46.566348] I [socket.c:3576:socket_init] 0-management: using system polling thread
[2015-02-26 00:14:46.566832] I [glusterd.c:138:glusterd_uuid_init] 0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option transport.socket.listen-backlog 128
  7:     option transport.socket.read-fail-log off
  8:     option transport.socket.keepalive-interval 2
  9:     option transport.socket.keepalive-time 10
 10:     option transport-type rdma
 11:     option working-directory /var/lib/glusterd
 12: end-volume
 13: 
+------------------------------------------------------------------------------+
[2015-02-26 00:14:46.570634] E [socket.c:2161:socket_connect_finish] 0-management: connection to 172.20.20.22:24007 failed (Connection refused)
[2015-02-26 00:14:46.571244] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2
[2015-02-26 00:14:47.001039] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:47.001183] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to hpc1 (0), ret: 0
[2015-02-26 00:14:47.015772] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1
[2015-02-26 00:14:47.015820] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2
[2015-02-26 00:14:47.030090] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-02-26 00:14:47.030178] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled
[2015-02-26 00:14:47.030196] I [socket.c:3576:socket_init] 0-management: using system polling thread
[2015-02-26 00:14:48.031850] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error: Permission denied
[2015-02-26 00:14:48.032392] I [glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully
[2015-02-26 00:14:48.032794] I [glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully
[2015-02-26 00:14:48.033182] I [glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully
[2015-02-26 00:14:48.033571] I [glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully
[2015-02-26 00:14:48.033964] I [glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully
[2015-02-26 00:14:48.034356] I [glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully
[2015-02-26 00:14:48.037872] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-02-26 00:14:48.037982] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled
[2015-02-26 00:14:48.038001] I [socket.c:3576:socket_init] 0-management: using system polling thread
[2015-02-26 00:14:49.039002] E [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management: Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No such file or directory
[2015-02-26 00:14:49.042564] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-02-26 00:14:49.042670] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled
[2015-02-26 00:14:49.042690] I [socket.c:3576:socket_init] 0-management: using system polling thread
[2015-02-26 00:14:49.042958] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:49.042998] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1
[2015-02-26 00:14:49.043013] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
[2015-02-26 00:14:49.043319] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0
[2015-02-26 00:14:49.204554] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now
[2015-02-26 00:14:49.204657] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now
[2015-02-26 00:14:49.204717] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:49.814028] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /export/sda/brick on port 49153
[2015-02-26 00:14:53.678347] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2
[2015-02-26 00:14:53.848276] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c
[2015-02-26 00:14:53.848407] I [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to gfs2 (0), ret: 0
[2015-02-26 00:14:53.862892] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1
[2015-02-26 00:14:53.862952] I [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2
[2015-02-26 00:14:53.877685] I [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:55.217486] I [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0
[2015-02-26 00:14:55.234370] I [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c
[2015-02-26 00:14:55.234433] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1
[2015-02-26 00:14:55.234451] I [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my uuid as Friend
[2015-02-26 00:14:55.234468] I [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1
[2015-02-26 00:15:14.637613] E [glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management: Volume gfsvolume already started
[2015-02-26 00:15:14.637661] E [glusterd-syncop.c:912:gd_stage_op_phase] 0-management: Staging of operation 'Volume Start' failed on localhost : Volume gfsvolume already started
[2015-02-26 00:24:41.265078] W [socket.c:522:__socket_rwv] 0-management: readv on 172.20.20.22:24007 failed (No data available)
[2015-02-26 00:24:43.631739] E [socket.c:2161:socket_connect_finish] 0-management: connection to 172.20.20.22:24007 failed (Connection refused)
[2015-02-26 00:24:56.174790] W [socket.c:522:__socket_rwv] 0-management: readv on /var/run/91bfef953906a83fc14aa435a886ac4d.socket failed (No data available)
[2015-02-26 00:24:56.175299] I [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management: Disconnected from gfs1:/export/sda/brick
[2015-02-26 00:25:24.076560] W [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed
[2015-02-26 00:25:24.078109] I [glusterd-handler.c:3530:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gfsvolume
[2015-02-26 00:25:31.041343] E [glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management: Volume gfsvolume already started
[2015-02-26 00:25:31.041404] E [glusterd-syncop.c:912:gd_stage_op_phase] 0-management: Staging of operation 'Volume Start' failed on localhost : Volume gfsvolume already started
[2015-02-26 00:25:43.389260] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8f8cfa7e6d] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f8f8d655b50] (-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f8f8eee1d55]))) 0-: received signum (15), shutting down
[2015-02-26 00:25:45.437325] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0 (/usr/sbin/glusterd -p /var/run/glusterd.pid)
[2015-02-26 00:25:45.487307] I [glusterd.c:1148:init] 0-management: Using /var/lib/glusterd as working directory
[2015-02-26 00:25:45.488734] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled
[2015-02-26 00:25:45.488765] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread
[2015-02-26 00:25:45.489557] W [rdma.c:4194:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
[2015-02-26 00:25:45.489584] E [rdma.c:4482:init] 0-rdma.management: Failed to initialize IB Device
[2015-02-26 00:25:45.489597] E [rpc-transport.c:333:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2015-02-26 00:25:45.489668] W [rpcsvc.c:1483:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2015-02-26 00:25:45.489799] I [socket.c:3561:socket_init] 0-socket.management: SSL support is NOT enabled
[2015-02-26 00:25:45.489818] I [socket.c:3576:socket_init] 0-socket.management: using system polling thread
[2015-02-26 00:25:48.706235] I [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2



Please Do let me know if there is anything else i can provide to help.  Apologies if this is a simple fix but i have never found an answer after a week of searching. 

Thank You Kindly,
Kaamesh

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users
[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux