Hi guys,
We have rails app, which is using gluster for our distributed file system. The glusters servers are hosted independently as part of deal with other, we don't have any impact on them, we are connected o them by using gluster native client.
We tried to resolve this issue using help from the admins of the company that is hosting our gluster servers, but they say that's the client issue and we ran out of ideas how that's possible if we are not doing anything special here.
Information about independent gluster servers:
-version: 3.6.0.42.1
- They are using red hat
-They are enterprise so the are always using older versions
Our servers:
System version: Ubuntu 14.04
Our gluster client version: 3.6.2
The exact problem is that it often happens(couple times a week) that errors in gluster causes proceses to become zombies. It happens with our application server(unicorn), nginx and our crawling script that is run as daemon.
Our fstab file:
10.10.11.17:/drslk-prod /mnt/storage glusterfs defaults,_netdev,nobootwait,fetch-attempts=10 0 0
10.10.11.17:/drslk-backup /mnt/backup glusterfs defaults,_netdev,nobootwait,fetch-attempts=10 0 0
Logs from gluster:
2015-02-18 12:36:12.375695] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7fb41ddeada6] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb41d
bc1c7e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb41dbc1d8e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x82)[0x7fb41dbc3602] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc
_clnt_notify+0x48)[0x7fb41dbc3d98] ))))) 0-drslk-prod-client-10: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-02-18 12:36:12.361489 (xid=0x5d475da)
[2015-02-18 12:36:12.375765] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-10: remote operation failed: Transport endpoint is not connected. Path: /system/posts/00/00/71/77/59.jpg (2ad81c2b-a141-478d-9dd4-253345edbce
b)
[2015-02-18 12:36:12.376288] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7fb41ddeada6] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fb41d
bc1c7e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb41dbc1d8e] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x82)[0x7fb41dbc3602] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc
_clnt_notify+0x48)[0x7fb41dbc3d98] ))))) 0-drslk-prod-client-10: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-02-18 12:36:12.361858 (xid=0x5d475db)
[2015-02-18 12:36:12.376355] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-10: remote operation failed: Transport endpoint is not connected. Path: /system/posts/00/00/08 (f5c33a99-719e-4ea2-ad1f-33b893af103d)
[2015-02-18 12:36:12.376711] I [socket.c:3292:socket_submit_request] 0-drslk-prod-client-10: not connected (priv->connected = 0)
[2015-02-18 12:36:12.376749] W [rpc-clnt.c:1562:rpc_clnt_submit] 0-drslk-prod-client-10: failed to submit rpc-request (XID: 0x5d475dc Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (drslk-prod-client-10)
[2015-02-18 12:36:12.376814] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-10: remote operation failed: Transport endpoint is not connected. Path: (null) (00000000-0000-0000-0000-000000000000)
[2015-02-18 12:36:12.376829] I [client.c:2215:client_rpc_notify] 0-drslk-prod-client-10: disconnected from drslk-prod-client-10. Client process will keep trying to connect to glusterd until brick's port is available
[2015-02-18 12:36:12.376834] W [rpc-clnt.c:1562:rpc_clnt_submit] 0-drslk-prod-client-10: failed to submit rpc-request (XID: 0x5d475dd Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (drslk-prod-client-10)
[2015-02-18 12:36:12.376906] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-10: remote operation failed: Transport endpoint is not connected. Path: (null) (00000000-0000-0000-0000-000000000000)
[2015-02-18 12:36:12.376931] E [socket.c:2267:socket_connect_finish] 0-drslk-prod-client-10: connection to 10.10.11.23:24007 failed (Connection refused)
[2015-02-18 12:36:12.379296] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-10: remote operation failed: Transport endpoint is not connected. Path: (null) (00000000-0000-0000-0000-000000000000)
[2015-02-18 12:36:12.379700] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-10: remote operation failed: Transport endpoint is not connected. Path: (null) (00000000-0000-0000-0000-000000000000)
[2015-02-18 13:10:52.759736] E [client-handshake.c:1496:client_query_portmap_cbk] 0-drslk-prod-client-10: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2015-02-18 13:10:52.759796] I [client.c:2215:client_rpc_notify] 0-drslk-prod-client-10: disconnected from drslk-prod-client-10. Client process will keep trying to connect to glusterd until brick's port is available
[2015-02-18 13:11:02.897307] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-10: changing port to 49349 (from 0)
[2015-02-18 13:11:02.898097] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-02-18 13:11:02.898446] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-10: Connected to drslk-prod-client-10, attached to remote volume '/GLUSTERFS/drslk-prod'.
[2015-02-18 13:11:02.898460] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-10: Server and Client lk-version numbers are not same, reopening the fds
Additional logs in attachments.
Did anyone encounter similiar issue with gluster? Do you have any ideas how to solve the problem?
Best regards,
Przemek
[2015-03-06 05:35:06.450939] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-3: remote operation failed: No data available [2015-03-06 05:35:06.451020] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-4: remote operation failed: No data available [2015-03-06 05:35:06.451052] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-5: remote operation failed: No data available [2015-03-06 05:35:06.451489] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 80195806: REMOVEXATTR() /system/posts/00/00/75/60/90.jpg => -1 (No data available) [2015-03-06 05:35:06.477557] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-2: remote operation failed: No data available [2015-03-06 05:35:06.477622] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-1: remote operation failed: No data available [2015-03-06 05:35:06.478055] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 80195818: REMOVEXATTR() /system/posts/00/00/75/60/90small.jpg => -1 (No data available) [2015-03-06 05:35:06.549543] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-3: remote operation failed: No data available [2015-03-06 05:35:06.549586] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-4: remote operation failed: No data available [2015-03-06 05:35:06.549613] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-5: remote operation failed: No data available [2015-03-06 05:35:06.550044] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 80195830: REMOVEXATTR() /system/posts/00/00/75/60/90medium.jpg => -1 (No data available) [2015-03-06 05:40:37.632329] E [client-rpc-fops.c:449:client3_3_open_cbk] 0-drslk-prod-client-1: remote operation failed: Permission denied. Path: <gfid:50bfa183-fef2-44f3-9c22-61d547161005> (50bfa183-fef2-44f3-9c22-61d547161005) [2015-03-06 05:40:37.632385] E [client-rpc-fops.c:449:client3_3_open_cbk] 0-drslk-prod-client-2: remote operation failed: Permission denied. Path: <gfid:50bfa183-fef2-44f3-9c22-61d547161005> (50bfa183-fef2-44f3-9c22-61d547161005) [2015-03-06 05:40:37.633522] I [afr-self-heal-metadata.c:54:__afr_selfheal_metadata_do] 0-drslk-prod-replicate-0: performing metadata selfheal on 50bfa183-fef2-44f3-9c22-61d547161005 [2015-03-06 05:40:37.634592] W [client-rpc-fops.c:2139:client3_3_setattr_cbk] 0-drslk-prod-client-2: remote operation failed: Operation not permitted [2015-03-06 05:40:37.635589] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-2: remote operation failed: Permission denied [2015-03-06 05:40:37.636063] W [client-rpc-fops.c:1023:client3_3_setxattr_cbk] 0-drslk-prod-client-2: remote operation failed: Permission denied [2015-03-06 05:40:37.637687] I [afr-self-heal-common.c:476:afr_log_selfheal] 0-drslk-prod-replicate-0: Completed metadata selfheal on 50bfa183-fef2-44f3-9c22-61d547161005. source=1 sinks= [2015-03-06 06:11:53.537870] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-0: changing port to 49207 (from 0) [2015-03-06 06:11:53.538803] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-03-06 06:11:53.539274] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-0: Connected to drslk-prod-client-0, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-03-06 06:11:53.539292] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-03-06 06:11:53.539448] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-0: Server lk version = 1 [2015-03-06 08:33:15.105174] E [rpc-clnt.c:201:call_bail] 0-drslk-prod-client-0: bailing out frame type(GlusterFS 3.3) op(LOOKUP(27)) xid = 0x41c011c sent = 2015-03-06 08:32:07.662018. timeout = 60 for 10.10.11.13:49207 [2015-03-06 08:33:15.105241] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-0: remote operation failed: Transport endpoint is not connected. Path: /system/posts/00/00/47/85/80.jpg (4df0c04c-7f59-465d-84f9-204aeb5a4fd9) [2015-03-06 08:33:25.341916] C [rpc-clnt.c:452:rpc_clnt_fill_request_info] 0-drslk-prod-client-0: cannot lookup the saved frame corresponding to xid (68944156) [2015-03-06 08:33:25.341979] W [socket.c:1892:__socket_read_reply] 0-drslk-prod-client-0: notify for event MAP_XID failed [2015-03-06 08:33:25.342214] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7fa1434392e6] (--> /usr/local/lib/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa14320cd5e] (--> /usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa14320ce6e] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x82)[0x7fa14320e6e2] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa14320ee78] ))))) 0-drslk-prod-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-03-06 08:33:15.105573 (xid=0x41c2e79) [2015-03-06 08:33:25.342240] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-0: remote operation failed: Transport endpoint is not connected. Path: (null) (00000000-0000-0000-0000-000000000000) [2015-03-06 08:33:25.342369] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7fa1434392e6] (--> /usr/local/lib/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa14320cd5e] (--> /usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa14320ce6e] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x82)[0x7fa14320e6e2] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa14320ee78] ))))) 0-drslk-prod-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-03-06 08:33:23.553928 (xid=0x41c3561) [2015-03-06 08:33:25.342403] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-0: remote operation failed: Transport endpoint is not connected. Path: /system/posts/00/00/21/57/16small. (19c88935-be14-4112-9935-bc675d374136) [2015-03-06 08:33:25.342653] I [socket.c:3292:socket_submit_request] 0-drslk-prod-client-0: not connected (priv->connected = 0) [2015-03-06 08:33:25.342673] W [rpc-clnt.c:1562:rpc_clnt_submit] 0-drslk-prod-client-0: failed to submit rpc-request (XID: 0x41c3701 Program: GlusterFS 3.3, ProgVers: 330, Proc: 31) to rpc-transport (drslk-prod-client-0) [2015-03-06 08:33:25.342689] E [client-rpc-fops.c:1621:client3_3_entrylk_cbk] 0-drslk-prod-client-0: remote operation failed: Transport endpoint is not connected [2015-03-06 08:33:25.342732] W [rpc-clnt.c:1562:rpc_clnt_submit] 0-drslk-prod-client-0: failed to submit rpc-request (XID: 0x41c3702 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (drslk-prod-client-0) [2015-03-06 08:33:25.342761] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-0: remote operation failed: Transport endpoint is not connected. Path: (null) (00000000-0000-0000-0000-000000000000) [2015-03-06 08:33:25.342816] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7fa1434392e6] (--> /usr/local/lib/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fa14320cd5e] (--> /usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fa14320ce6e] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x82)[0x7fa14320e6e2] (--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x48)[0x7fa14320ee78] ))))) 0-drslk-prod-client-0: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2015-03-06 08:33:24.813086 (xid=0x41c36ca) [2015-03-06 08:33:25.342840] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-0: remote operation failed: Transport endpoint is not connected. Path: /system/posts/00/00/22/32/30. (00000000-0000-0000-0000-000000000000) [2015-03-06 08:33:25.343109] I [client.c:2215:client_rpc_notify] 0-drslk-prod-client-0: disconnected from drslk-prod-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-03-06 08:33:25.343559] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-0: changing port to 49207 (from 0) [2015-03-06 08:33:25.343674] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-0: remote operation failed: Transport endpoint is not connected. Path: (null) (00000000-0000-0000-0000-000000000000) [2015-03-06 08:33:25.440212] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-03-06 08:33:25.440374] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-0: remote operation failed: Transport endpoint is not connected. Path: (null) (00000000-0000-0000-0000-000000000000) [2015-03-06 08:33:25.440440] W [client-rpc-fops.c:2766:client3_3_lookup_cbk] 0-drslk-prod-client-0: remote operation failed: Transport endpoint is not connected. Path: (null) (00000000-0000-0000-0000-000000000000) [2015-03-06 08:33:25.440637] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-0: Connected to drslk-prod-client-0, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-03-06 08:33:25.440674] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-03-06 08:33:25.441136] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-0: Server lk version = 1 [2015-03-06 08:59:45.491056] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-3: remote operation failed: No data available [2015-03-06 08:59:45.491143] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-4: remote operation failed: No data available [2015-03-06 08:59:45.491189] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-5: remote operation failed: No data available [2015-03-06 08:59:45.491536] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 82939089: REMOVEXATTR() /system/avatars/00/00/03/24/91.png => -1 (No data available) [2015-03-06 08:59:45.529396] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-0: remote operation failed: No data available [2015-03-06 08:59:45.529459] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-1: remote operation failed: No data available [2015-03-06 08:59:45.529492] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-2: remote operation failed: No data available [2015-03-06 08:59:45.529898] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 82939101: REMOVEXATTR() /system/avatars/00/00/03/24/91small.png => -1 (No data available) [2015-03-06 08:59:45.539722] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-4: remote operation failed: No data available [2015-03-06 08:59:45.539780] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-5: remote operation failed: No data available [2015-03-06 08:59:45.539813] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-3: remote operation failed: No data available [2015-03-06 08:59:45.540187] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 82939113: REMOVEXATTR() /system/avatars/00/00/03/24/91medium.png => -1 (No data available) [2015-03-06 08:59:45.550851] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-0: remote operation failed: No data available [2015-03-06 08:59:45.550881] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-1: remote operation failed: No data available [2015-03-06 08:59:45.550921] W [client-rpc-fops.c:1210:client3_3_removexattr_cbk] 0-drslk-prod-client-2: remote operation failed: No data available [2015-03-06 08:59:45.551254] W [fuse-bridge.c:1261:fuse_err_cbk] 0-glusterfs-fuse: 82939127: REMOVEXATTR() /system/avatars/00/00/03/24/91big.png => -1 (No data available)
[2015-02-23 13:23:04.230411] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed [2015-02-23 13:23:04.860446] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed [2015-02-23 13:23:04.913919] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2015-02-23 13:23:05.064055] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2015-02-23 13:23:18.436250] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed [2015-02-23 13:23:18.544791] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed [2015-02-23 13:23:18.574771] I [dht-shared.c:337:dht_init_regex] 0-drslk-prod-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ [2015-02-23 13:23:18.598064] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2015-02-23 13:51:51.071115] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed [2015-02-23 13:51:51.166100] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed [2015-02-23 13:51:51.195595] I [dht-shared.c:337:dht_init_regex] 0-drslk-prod-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ [2015-02-23 13:51:51.220195] I [glusterfsd-mgmt.c:1504:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2015-02-23 16:19:32.237944] W [socket.c:611:__socket_rwv] 0-drslk-prod-client-11: readv on 10.10.11.24:49383 failed (No data available) [2015-02-23 16:19:32.238051] I [client.c:2215:client_rpc_notify] 0-drslk-prod-client-11: disconnected from drslk-prod-client-11. Client process will keep trying to connect to glusterd until brick's port is available [2015-02-23 16:19:42.471876] E [socket.c:2267:socket_connect_finish] 0-drslk-prod-client-11: connection to 10.10.11.24:24007 failed (Connection refused) [2015-02-23 16:21:14.713977] E [client-handshake.c:1496:client_query_portmap_cbk] 0-drslk-prod-client-11: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-02-23 16:21:14.714088] I [client.c:2215:client_rpc_notify] 0-drslk-prod-client-11: disconnected from drslk-prod-client-11. Client process will keep trying to connect to glusterd until brick's port is available [2015-02-23 16:21:35.621187] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-11: changing port to 49383 (from 0) [2015-02-23 16:21:35.622346] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-02-23 16:21:35.624794] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-11: Connected to drslk-prod-client-11, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-02-23 16:21:35.624832] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-11: Server and Client lk-version numbers are not same, reopening the fds [2015-02-23 16:21:35.624847] I [client-handshake.c:1052:client_post_handshake] 0-drslk-prod-client-11: 1 fds open - Delaying child_up until they are re-opened [2015-02-23 16:21:36.300272] W [client-handshake.c:807:client3_3_reopendir_cbk] 0-drslk-prod-client-11: reopendir on <gfid:fba0e4a5-8bb6-4d7f-9efb-25ea7902232f> failed (Stale file handle) [2015-02-23 16:21:36.300317] I [client-handshake.c:674:client_child_up_reopen_done] 0-drslk-prod-client-11: last fd open'd/lock-self-heal'd - notifying CHILD-UP [2015-02-23 16:21:36.300683] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-11: Server lk version = 1 [2015-02-23 21:12:33.099516] W [glusterfsd.c:1194:cleanup_and_exit] (--> 0-: received signum (15), shutting down [2015-02-23 21:12:33.099550] I [fuse-bridge.c:5599:fini] 0-fuse: Unmounting '/mnt/storage'. [2015-02-23 21:15:00.128406] I [MSGID: 100030] [glusterfsd.c:2018:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.2 (args: /usr/sbin/glusterfs --volfile-server=10.10.11.13 --volfile-id=/drslk-prod /mnt/storage) [2015-02-23 21:15:00.142079] I [dht-shared.c:337:dht_init_regex] 0-drslk-prod-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$ [2015-02-23 21:15:00.147922] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-drslk-prod-client-11: setting frame-timeout to 60 [2015-02-23 21:15:00.148138] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-drslk-prod-client-10: setting frame-timeout to 60 [2015-02-23 21:15:00.148336] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-drslk-prod-client-9: setting frame-timeout to 60 [2015-02-23 21:15:00.148532] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-drslk-prod-client-8: setting frame-timeout to 60 [2015-02-23 21:15:00.148725] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-drslk-prod-client-7: setting frame-timeout to 60 [2015-02-23 21:15:00.148920] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-drslk-prod-client-6: setting frame-timeout to 60 [2015-02-23 21:15:00.149114] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-drslk-prod-client-5: setting frame-timeout to 60 [2015-02-23 21:15:00.149309] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-drslk-prod-client-4: setting frame-timeout to 60 [2015-02-23 21:15:00.149523] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-drslk-prod-client-3: setting frame-timeout to 60 [2015-02-23 21:15:00.149721] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-drslk-prod-client-2: setting frame-timeout to 60 [2015-02-23 21:15:00.149916] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-drslk-prod-client-1: setting frame-timeout to 60 [2015-02-23 21:15:00.150110] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-drslk-prod-client-0: setting frame-timeout to 60 [2015-02-23 21:15:00.150298] I [client.c:2280:notify] 0-drslk-prod-client-0: parent translators are ready, attempting connect on transport [2015-02-23 21:15:00.150633] I [client.c:2280:notify] 0-drslk-prod-client-1: parent translators are ready, attempting connect on transport [2015-02-23 21:15:00.150897] I [client.c:2280:notify] 0-drslk-prod-client-2: parent translators are ready, attempting connect on transport [2015-02-23 21:15:00.151105] I [client.c:2280:notify] 0-drslk-prod-client-3: parent translators are ready, attempting connect on transport [2015-02-23 21:15:00.151304] I [client.c:2280:notify] 0-drslk-prod-client-4: parent translators are ready, attempting connect on transport [2015-02-23 21:15:00.151511] I [client.c:2280:notify] 0-drslk-prod-client-5: parent translators are ready, attempting connect on transport [2015-02-23 21:15:00.151693] I [client.c:2280:notify] 0-drslk-prod-client-6: parent translators are ready, attempting connect on transport [2015-02-23 21:15:00.151898] I [client.c:2280:notify] 0-drslk-prod-client-7: parent translators are ready, attempting connect on transport [2015-02-23 21:15:00.152110] I [client.c:2280:notify] 0-drslk-prod-client-8: parent translators are ready, attempting connect on transport [2015-02-23 21:15:00.152322] I [client.c:2280:notify] 0-drslk-prod-client-9: parent translators are ready, attempting connect on transport [2015-02-23 21:15:00.152535] I [client.c:2280:notify] 0-drslk-prod-client-10: parent translators are ready, attempting connect on transport [2015-02-23 21:15:00.152723] I [client.c:2280:notify] 0-drslk-prod-client-11: parent translators are ready, attempting connect on transport Final graph: +------------------------------------------------------------------------------+ 1: volume drslk-prod-client-0 2: type protocol/client 3: option ping-timeout 20 4: option remote-host brick13.gluster.iadm 5: option remote-subvolume /GLUSTERFS/drslk-prod 6: option transport-type socket 7: option frame-timeout 60 8: option send-gids true 9: end-volume 10: 11: volume drslk-prod-client-1 12: type protocol/client 13: option ping-timeout 20 14: option remote-host brick14.gluster.iadm 15: option remote-subvolume /GLUSTERFS/drslk-prod 16: option transport-type socket 17: option frame-timeout 60 18: option send-gids true 19: end-volume 20: 21: volume drslk-prod-client-2 22: type protocol/client 23: option ping-timeout 20 24: option remote-host brick15.gluster.iadm 25: option remote-subvolume /GLUSTERFS/drslk-prod 26: option transport-type socket 27: option frame-timeout 60 28: option send-gids true 29: end-volume 30: 31: volume drslk-prod-replicate-0 32: type cluster/replicate 33: option read-hash-mode 2 34: option data-self-heal-window-size 128 35: option quorum-type auto 36: subvolumes drslk-prod-client-0 drslk-prod-client-1 drslk-prod-client-2 37: end-volume 38: 39: volume drslk-prod-client-3 40: type protocol/client 41: option ping-timeout 20 42: option remote-host brick16.gluster.iadm 43: option remote-subvolume /GLUSTERFS/drslk-prod 44: option transport-type socket 45: option frame-timeout 60 46: option send-gids true 47: end-volume 48: 49: volume drslk-prod-client-4 50: type protocol/client 51: option ping-timeout 20 52: option remote-host brick17.gluster.iadm 53: option remote-subvolume /GLUSTERFS/drslk-prod 54: option transport-type socket 55: option frame-timeout 60 56: option send-gids true 57: end-volume 58: 59: volume drslk-prod-client-5 60: type protocol/client 61: option ping-timeout 20 62: option remote-host brick18.gluster.iadm 63: option remote-subvolume /GLUSTERFS/drslk-prod 64: option transport-type socket 65: option frame-timeout 60 66: option send-gids true 67: end-volume 68: 69: volume drslk-prod-replicate-1 70: type cluster/replicate 71: option read-hash-mode 2 72: option data-self-heal-window-size 128 73: option quorum-type auto 74: subvolumes drslk-prod-client-3 drslk-prod-client-4 drslk-prod-client-5 75: end-volume 76: 77: volume drslk-prod-client-6 78: type protocol/client 79: option ping-timeout 20 80: option remote-host brick19.gluster.iadm 81: option remote-subvolume /GLUSTERFS/drslk-prod 82: option transport-type socket 83: option frame-timeout 60 84: option send-gids true 85: end-volume 86: 87: volume drslk-prod-client-7 88: type protocol/client 89: option ping-timeout 20 90: option remote-host brick20.gluster.iadm 91: option remote-subvolume /GLUSTERFS/drslk-prod 92: option transport-type socket 93: option frame-timeout 60 94: option send-gids true 95: end-volume 96: 97: volume drslk-prod-client-8 98: type protocol/client 99: option ping-timeout 20 100: option remote-host brick21.gluster.iadm 101: option remote-subvolume /GLUSTERFS/drslk-prod 102: option transport-type socket 103: option frame-timeout 60 104: option send-gids true 105: end-volume 106: 107: volume drslk-prod-replicate-2 108: type cluster/replicate 109: option read-hash-mode 2 110: option data-self-heal-window-size 128 111: option quorum-type auto 112: subvolumes drslk-prod-client-6 drslk-prod-client-7 drslk-prod-client-8 113: end-volume 114: 115: volume drslk-prod-client-9 116: type protocol/client 117: option ping-timeout 20 118: option remote-host brick22.gluster.iadm 119: option remote-subvolume /GLUSTERFS/drslk-prod 120: option transport-type socket 121: option frame-timeout 60 122: option send-gids true 123: end-volume 124: 125: volume drslk-prod-client-10 126: type protocol/client 127: option ping-timeout 20 128: option remote-host brick23.gluster.iadm 129: option remote-subvolume /GLUSTERFS/drslk-prod 130: option transport-type socket 131: option frame-timeout 60 132: option send-gids true 133: end-volume 134: 135: volume drslk-prod-client-11 136: type protocol/client 137: option ping-timeout 20 138: option remote-host brick24.gluster.iadm 139: option remote-subvolume /GLUSTERFS/drslk-prod 140: option transport-type socket 141: option frame-timeout 60 142: option send-gids true 143: end-volume 144: 145: volume drslk-prod-replicate-3 146: type cluster/replicate 147: option read-hash-mode 2 148: option data-self-heal-window-size 128 149: option quorum-type auto 150: subvolumes drslk-prod-client-9 drslk-prod-client-10 drslk-prod-client-11 151: end-volume 152: 153: volume drslk-prod-dht 154: type cluster/distribute 155: option min-free-disk 10% 156: option readdir-optimize on 157: subvolumes drslk-prod-replicate-0 drslk-prod-replicate-1 drslk-prod-replicate-2 drslk-prod-replicate-3 158: end-volume 159: 160: volume drslk-prod-write-behind 161: type performance/write-behind 162: option cache-size 1MB 163: subvolumes drslk-prod-dht 164: end-volume 165: 166: volume drslk-prod-read-ahead 167: type performance/read-ahead 168: subvolumes drslk-prod-write-behind 169: end-volume 170: 171: volume drslk-prod-readdir-ahead 172: type performance/readdir-ahead 173: subvolumes drslk-prod-read-ahead 174: end-volume 175: 176: volume drslk-prod-io-cache 177: type performance/io-cache 178: option cache-timeout 60 179: option cache-size 512MB 180: subvolumes drslk-prod-readdir-ahead 181: end-volume 182: 183: volume drslk-prod-quick-read 184: type performance/quick-read 185: option cache-size 512MB 186: subvolumes drslk-prod-io-cache 187: end-volume 188: 189: volume drslk-prod-md-cache 190: type performance/md-cache 191: subvolumes drslk-prod-quick-read 192: end-volume 193: 194: volume drslk-prod 195: type debug/io-stats 196: option latency-measurement off 197: option count-fop-hits off 198: subvolumes drslk-prod-md-cache 199: end-volume 200: 201: volume meta-autoload 202: type meta 203: subvolumes drslk-prod 204: end-volume 205: +------------------------------------------------------------------------------+ [2015-02-23 21:15:00.153792] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-0: changing port to 49207 (from 0) [2015-02-23 21:15:00.153815] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-1: changing port to 49371 (from 0) [2015-02-23 21:15:00.154260] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-5: changing port to 49356 (from 0) [2015-02-23 21:15:00.154282] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-4: changing port to 49284 (from 0) [2015-02-23 21:15:00.154676] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-9: changing port to 49320 (from 0) [2015-02-23 21:15:00.154699] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-8: changing port to 49281 (from 0) [2015-02-23 21:15:00.154715] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-2: changing port to 49327 (from 0) [2015-02-23 21:15:00.154734] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-3: changing port to 49279 (from 0) [2015-02-23 21:15:00.154750] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-7: changing port to 49300 (from 0) [2015-02-23 21:15:00.154764] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-6: changing port to 49398 (from 0) [2015-02-23 21:15:00.155923] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-11: changing port to 49383 (from 0) [2015-02-23 21:15:00.156198] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-02-23 21:15:00.156242] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-02-23 21:15:00.156356] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-02-23 21:15:00.156389] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-02-23 21:15:00.156435] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-9: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-02-23 21:15:00.156469] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-8: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-02-23 21:15:00.156509] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-0: Connected to drslk-prod-client-0, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-02-23 21:15:00.156526] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-02-23 21:15:00.156558] I [MSGID: 108005] [afr-common.c:3552:afr_notify] 0-drslk-prod-replicate-0: Subvolume 'drslk-prod-client-0' came back up; going online. [2015-02-23 21:15:00.156587] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-1: Connected to drslk-prod-client-1, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-02-23 21:15:00.156595] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-02-23 21:15:00.156637] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-02-23 21:15:00.156670] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-7: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-02-23 21:15:00.156702] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-02-23 21:15:00.156731] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-02-23 21:15:00.156765] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-1: Server lk version = 1 [2015-02-23 21:15:00.156778] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-11: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-02-23 21:15:00.156812] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-4: Connected to drslk-prod-client-4, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-02-23 21:15:00.156819] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-4: Server and Client lk-version numbers are not same, reopening the fds [2015-02-23 21:15:00.156847] I [MSGID: 108005] [afr-common.c:3552:afr_notify] 0-drslk-prod-replicate-1: Subvolume 'drslk-prod-client-4' came back up; going online. [2015-02-23 21:15:00.156865] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-9: Connected to drslk-prod-client-9, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-02-23 21:15:00.156871] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-9: Server and Client lk-version numbers are not same, reopening the fds [2015-02-23 21:15:00.156899] I [MSGID: 108005] [afr-common.c:3552:afr_notify] 0-drslk-prod-replicate-3: Subvolume 'drslk-prod-client-9' came back up; going online. [2015-02-23 21:15:00.156916] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-5: Connected to drslk-prod-client-5, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-02-23 21:15:00.156922] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-5: Server and Client lk-version numbers are not same, reopening the fds [2015-02-23 21:15:00.156953] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-0: Server lk version = 1 [2015-02-23 21:15:00.156966] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-8: Connected to drslk-prod-client-8, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-02-23 21:15:00.156972] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-8: Server and Client lk-version numbers are not same, reopening the fds [2015-02-23 21:15:00.156996] I [MSGID: 108005] [afr-common.c:3552:afr_notify] 0-drslk-prod-replicate-2: Subvolume 'drslk-prod-client-8' came back up; going online. [2015-02-23 21:15:00.157022] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-4: Server lk version = 1 [2015-02-23 21:15:00.157039] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-9: Server lk version = 1 [2015-02-23 21:15:00.157079] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-2: Connected to drslk-prod-client-2, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-02-23 21:15:00.157090] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-2: Server and Client lk-version numbers are not same, reopening the fds [2015-02-23 21:15:00.157175] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-7: Connected to drslk-prod-client-7, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-02-23 21:15:00.157184] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-7: Server and Client lk-version numbers are not same, reopening the fds [2015-02-23 21:15:00.157215] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-3: Connected to drslk-prod-client-3, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-02-23 21:15:00.157222] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-3: Server and Client lk-version numbers are not same, reopening the fds [2015-02-23 21:15:00.157286] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-6: Connected to drslk-prod-client-6, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-02-23 21:15:00.157297] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-6: Server and Client lk-version numbers are not same, reopening the fds [2015-02-23 21:15:00.157379] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-2: Server lk version = 1 [2015-02-23 21:15:00.157394] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-11: Connected to drslk-prod-client-11, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-02-23 21:15:00.157400] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-11: Server and Client lk-version numbers are not same, reopening the fds [2015-02-23 21:15:00.157437] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-5: Server lk version = 1 [2015-02-23 21:15:00.157451] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-8: Server lk version = 1 [2015-02-23 21:15:00.157491] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-7: Server lk version = 1 [2015-02-23 21:15:00.157503] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-3: Server lk version = 1 [2015-02-23 21:15:00.157557] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-6: Server lk version = 1 [2015-02-23 21:15:00.157588] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-11: Server lk version = 1 [2015-02-23 21:15:10.127369] I [rpc-clnt.c:1761:rpc_clnt_reconfig] 0-drslk-prod-client-10: changing port to 49349 (from 0) [2015-02-23 21:15:10.128141] I [client-handshake.c:1413:select_server_supported_programs] 0-drslk-prod-client-10: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-02-23 21:15:10.128520] I [client-handshake.c:1200:client_setvolume_cbk] 0-drslk-prod-client-10: Connected to drslk-prod-client-10, attached to remote volume '/GLUSTERFS/drslk-prod'. [2015-02-23 21:15:10.128534] I [client-handshake.c:1210:client_setvolume_cbk] 0-drslk-prod-client-10: Server and Client lk-version numbers are not same, reopening the fds [2015-02-23 21:15:10.130828] I [fuse-bridge.c:5080:fuse_graph_setup] 0-fuse: switched to graph 0 [2015-02-23 21:15:10.130910] I [client-handshake.c:188:client_set_lk_version_cbk] 0-drslk-prod-client-10: Server lk version = 1 [2015-02-23 21:15:10.131056] I [fuse-bridge.c:4009:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
