Hello,
We are running an oVirt cluster on top of a 2 node replica 2 Gluster volume. Yesterday we suddenly noticed VMs were not responding and quickly found out the Gluster volume had issues. These errors were filling up the etc-glusterfs-glusterd.log file:
[2015-06-07 08:36:26.498012] W [rpcsvc.c:270:rpcsvc_program_actor] 0-rpc-service: RPC program not available (req 1298437 330) for 10.100.3.41:1022
[2015-06-07 08:36:26.498073] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
A restart of glusterfsd and glusterd resolved the issue, but triggered a lot of self-heals.
We are running glusterfs 3.7.0 on ZFS.
I have attached etc-glusterfs-glusterd.log, the brick log file and the glustershd.log. I would be grateful if anyone could shed any light on what happened here and if there's anything we can do to prevent it.
--
Tiemen Ruiten
Systems Engineer
R&D Media
Systems Engineer
R&D Media
Attachment:
etc-glusterfs-glusterd.vol.log-20150608
Description: Binary data
[2015-06-07 01:46:02.388793] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2015-06-07 08:36:27.574672] W [socket.c:642:__socket_rwv] 0-tcp.vmimage-server: writev on 10.100.3.41:1020 failed (Broken pipe)
[2015-06-07 08:36:27.679161] I [socket.c:2409:socket_event_handler] 0-transport: disconnecting now
[2015-06-07 08:36:27.681969] I [server.c:530:server_rpc_notify] 0-vmimage-server: disconnecting connection from dibbler.int.rdmedia.com-25881-2015/05/21-04:29:49:844757-vmimage-client-0-0-0
[2015-06-07 08:36:27.695294] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-vmimage-server: releasing lock on 950a47c7-5d0e-457f-a6da-61d2c2ba0e1c held by {client=0x7f7a180009e0, pid=0 lk-owner=8099089cbf7f0000}
[2015-06-07 08:36:27.695325] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-vmimage-server: releasing lock on 3186cd00-e203-4bc5-890e-dc026d60a61c held by {client=0x7f7a180009e0, pid=0 lk-owner=e09c089cbf7f0000}
[2015-06-07 08:36:27.695340] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-vmimage-server: releasing lock on f90e8747-ca3e-42fa-940f-3d82353c596b held by {client=0x7f7a180009e0, pid=0 lk-owner=249e089cbf7f0000}
[2015-06-07 08:36:27.695352] W [inodelk.c:404:pl_inodelk_log_cleanup] 0-vmimage-server: releasing lock on 942a9620-90ce-48d6-9ea5-5a0972dc2e7e held by {client=0x7f7a180009e0, pid=0 lk-owner=b89d089cbf7f0000}
[2015-06-07 08:36:27.696747] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/dom_md/ids
[2015-06-07 08:36:27.696808] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/73f90643-4310-4929-9a4d-559584c562ee/114ab2cc-40c0-44ab-8ab4-f8abf7309d4c
[2015-06-07 08:36:27.696839] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/aea4b8f7-1d05-4187-8695-81c16c92cfff/b64055dd-a6dd-40da-8260-4d9877e39e97
[2015-06-07 08:36:27.696868] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/745eb29d-8e3f-4746-8bc5-7c143107a038/c944ffbe-2451-40e9-a6e2-bd2b5d9b38ae
[2015-06-07 08:36:27.696914] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/c1bb0013-fbd9-4b3a-9da4-eca44725842f/db57af32-fdb5-4efb-ab54-7630dceac428
[2015-06-07 08:36:27.696935] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/01ca6661-0410-45aa-b587-e03ca0990bd3/9c0e8c59-8a95-48e8-85d8-47df9000c48b
[2015-06-07 08:36:27.696954] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/6d8dcb56-429d-4bd3-832e-a6d619d9cba6/993b1f8b-9c90-4799-b1df-64ef474f4e53
[2015-06-07 08:36:27.696981] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/4495a54a-dbdf-425a-9f82-7c7732e23fd6/011acee0-eba4-4267-a090-5008f8a6a82d
[2015-06-07 08:36:27.697002] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/59746d12-537f-4a89-a8e1-21fee68119f2/68d6658b-7387-4cef-a27a-90252e8ff72c
[2015-06-07 08:36:27.697023] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/adc2b6cd-ac91-4e6a-8792-f74bfb247644/e72578d5-77e7-4bed-84a6-64e3383fe5a3
[2015-06-07 08:36:27.697057] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/dacb816f-3028-48c3-b912-56a458a7ef92/aa541399-fd15-4407-9cd2-a481377534d2
[2015-06-07 08:36:27.697075] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/464f88e5-826f-4eee-9db2-d0cd4f6d84b6/53dd33c3-e276-46b6-baa6-356d8ef78106
[2015-06-07 08:36:27.710562] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/8053eac3-bc5b-4952-b32d-f337db071d81/9d959474-56a4-4d4a-baec-0a7eaba3413c
[2015-06-07 08:36:27.710577] I [socket.c:3432:socket_submit_reply] 0-tcp.vmimage-server: not connected (priv->connected = -1)
[2015-06-07 08:36:27.710618] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/b24d5ccc-d149-43d9-9f53-f3951f32ddf3/dc28b5b9-b906-45cb-bd6e-0101349ead5e
[2015-06-07 08:36:27.710620] E [rpcsvc.c:1312:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0xa1108d8, Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (tcp.vmimage-server)
[2015-06-07 08:36:27.710636] I [server-helpers.c:290:do_fd_cleanup] 0-vmimage-server: fd cleanup on /732ea98e-5b18-4f68-8d1d-d0fb78c1b06d/images/a185c214-b3fb-4094-862c-fa1ae8759a76/21f8218e-453b-4a06-adbd-754e25cb80d3
[2015-06-07 08:36:27.711791] E [rpcsvc.c:1312:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0xa1108d9, Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (tcp.vmimage-server)
[2015-06-07 08:36:27.809391] E [server.c:196:server_submit_reply] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server_submit_reply+0x286)[0x7f7a1d6023c6] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server_writev_cbk+0xc2)[0x7f7a1d616b22] (--> /usr/lib64/glusterfs/3.7.0/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1de)[0x7f7a1d84042e] (--> /usr/lib64/glusterfs/3.7.0/xlator/features/marker.so(marker_writev_cbk+0xe7)[0x7f7a1e085ef7] ))))) 0-: Reply submission failed
[2015-06-07 08:36:27.809393] E [server.c:196:server_submit_reply] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server_submit_reply+0x286)[0x7f7a1d6023c6] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server_writev_cbk+0xc2)[0x7f7a1d616b22] (--> /usr/lib64/glusterfs/3.7.0/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1de)[0x7f7a1d84042e] (--> /usr/lib64/glusterfs/3.7.0/xlator/features/marker.so(marker_writev_cbk+0xe7)[0x7f7a1e085ef7] ))))) 0-: Reply submission failed
[2015-06-07 08:36:27.809486] I [client_t.c:401:gf_client_unref] 0-vmimage-server: Shutting down connection dibbler.int.rdmedia.com-25881-2015/05/21-04:29:49:844757-vmimage-client-0-0-0
[2015-06-07 08:36:27.809483] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.809533] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.809795] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.809818] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.809996] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.810022] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.810233] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.810255] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.810454] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.810489] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.810704] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.810727] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.810898] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.810919] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.811082] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.811104] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.811268] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.811294] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.811458] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.811486] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.811696] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.811729] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.811915] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.811937] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.812102] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.812123] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.812299] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.812321] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.812501] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.812524] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.812686] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.812707] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.812867] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.812890] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.813043] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.813064] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.813219] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.813241] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.813410] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.813431] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.813668] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.813691] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.813866] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.813892] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.814055] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.814076] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.814236] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.814258] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.814418] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.814440] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.814619] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.814641] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.814814] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.814852] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.815029] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.815057] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.815220] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.815243] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.815412] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.815434] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.815616] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.815639] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.815808] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.815830] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.816006] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.816027] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.816213] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.816239] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.816416] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.816438] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.816634] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.816669] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.816851] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.816877] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.817072] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.817093] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.817254] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.817275] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.817468] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.817503] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.817668] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.817691] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.817875] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.817897] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.818078] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.818099] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.818311] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.818333] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.818500] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.818522] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.818697] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.818720] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.818919] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.818941] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.819151] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.819176] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.819344] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.819367] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.819553] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.819576] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.819812] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.819834] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.820008] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.820029] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.820205] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.820231] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.820401] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.820423] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.820588] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.820611] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.820770] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.820801] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.820981] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.821002] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.821192] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.821213] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.821376] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.821401] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.821578] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.821611] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.821778] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.821800] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.821979] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.822001] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.822185] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.822208] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.822398] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.822420] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.822586] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.822617] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.822802] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.822825] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.822990] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.823011] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.823206] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.823227] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.823401] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.823422] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.823598] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.823626] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.823788] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.823813] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.824001] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.824023] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.824184] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.824205] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.824373] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.824395] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.824585] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.824608] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.824790] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.824812] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.824974] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.825022] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.825221] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.825243] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.825405] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_finodelk+0x97)[0x7f7a1d613217] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.825426] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.825604] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.825627] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.825817] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.825839] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.826012] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.826034] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.826194] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.826215] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.826372] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.826398] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.826570] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.826592] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.826753] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.826774] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.826940] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_writev+0x6d)[0x7f7a1d60c62d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.826962] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.827119] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_lookup+0x9d)[0x7f7a1d62038d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.827141] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.827340] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_writev+0x6d)[0x7f7a1d60c62d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.827362] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.827531] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_lookup+0x9d)[0x7f7a1d62038d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.827557] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.827752] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_writev+0x6d)[0x7f7a1d60c62d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.827774] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.827933] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_lookup+0x9d)[0x7f7a1d62038d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.827954] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.828120] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_writev+0x6d)[0x7f7a1d60c62d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.828142] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.828328] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_writev+0x6d)[0x7f7a1d60c62d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.828350] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.828531] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_lookup+0x9d)[0x7f7a1d62038d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.828556] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.828744] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_writev+0x6d)[0x7f7a1d60c62d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.828771] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.828963] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_writev+0x6d)[0x7f7a1d60c62d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.828987] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.829168] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_writev+0x6d)[0x7f7a1d60c62d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.829190] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.829354] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_lookup+0x9d)[0x7f7a1d62038d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.829374] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.829546] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_lookup+0x9d)[0x7f7a1d62038d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.829568] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.829727] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.829749] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.829906] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_fxattrop+0x99)[0x7f7a1d60b979] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.829949] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.830116] E [server-helpers.c:382:server_alloc_frame] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f7a31819fb0] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(get_frame_from_request+0x3a9)[0x7f7a1d607ad9] (--> /usr/lib64/glusterfs/3.7.0/xlator/protocol/server.so(server3_3_lookup+0x9d)[0x7f7a1d62038d] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f7a315e4db5] (--> /usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f7a315e4ff3] ))))) 0-server: invalid argument: client
[2015-06-07 08:36:27.830137] E [rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully
[2015-06-07 08:36:27.846461] I [server-handshake.c:585:server_setvolume] 0-vmimage-server: accepted client from dibbler.int.rdmedia.com-25881-2015/05/21-04:29:49:844757-vmimage-client-0-0-1 (version: 3.7.0)
[2015-06-07 08:36:27.954961] E [inodelk.c:309:__inode_unlock_lock] 0-vmimage-locks: Matching lock not found for unlock 0-9223372036854775807, by 249e089cbf7f0000 on 0x7f7a200ff5d0
[2015-06-07 08:36:27.955038] E [server-rpc-fops.c:279:server_finodelk_cbk] 0-vmimage-server: 168888986: FINODELK 12 (f90e8747-ca3e-42fa-940f-3d82353c596b) ==> (Invalid argument)
[2015-06-07 08:59:30.293283] W [glusterfsd.c:1219:cleanup_and_exit] (--> 0-: received signum (15), shutting down
[2015-06-07 09:01:04.247777] I [MSGID: 100030] [glusterfsd.c:2294:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.0 (args: /usr/sbin/glusterfsd -s 10.100.3.10 --volfile-id vmimage.10.100.3.10.export-gluster01-brick -p /var/lib/glusterd/vols/vmimage/run/10.100.3.10-export-gluster01-brick.pid -S /var/run/gluster/2bfe3a2242d586d0850775f601f1c3ee.socket --brick-name /export/gluster01/brick -l /var/log/glusterfs/bricks/export-gluster01-brick.log --xlator-option *-posix.glusterd-uuid=26186ec6-a8c7-4834-bcaa-24e30289dba3 --brick-port 49153 --xlator-option vmimage-server.listen-port=49153)
[2015-06-07 09:01:04.258754] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-06-07 09:01:04.460388] I [graph.c:269:gf_add_cmdline_options] 0-vmimage-server: adding option 'listen-port' for volume 'vmimage-server' with value '49153'
[2015-06-07 09:01:04.460454] I [graph.c:269:gf_add_cmdline_options] 0-vmimage-posix: adding option 'glusterd-uuid' for volume 'vmimage-posix' with value '26186ec6-a8c7-4834-bcaa-24e30289dba3'
[2015-06-07 09:01:04.460926] I [server.c:392:_check_for_auth_option] 0-/export/gluster01/brick: skip format check for non-addr auth option auth.login./export/gluster01/brick.allow
[2015-06-07 09:01:04.460936] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2015-06-07 09:01:04.460956] I [server.c:392:_check_for_auth_option] 0-/export/gluster01/brick: skip format check for non-addr auth option auth.login.86c3b46e-4b50-4e11-9ef4-cb8395a82c7d.password
[2015-06-07 09:01:04.462636] I [rpcsvc.c:2213:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64
[2015-06-07 09:01:04.462747] W [options.c:936:xl_opt_validate] 0-vmimage-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
[2015-06-07 09:01:04.465770] E [ctr-helper.c:250:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled.
[2015-06-07 09:01:04.465813] W [gfdb_sqlite3.h:238:gfdb_set_sql_params] 0-vmimage-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096
[2015-06-07 09:01:04.465833] W [gfdb_sqlite3.h:238:gfdb_set_sql_params] 0-vmimage-changetimerecorder: Failed to retrieve sql-db-cachesize from params.Assigning default value: 1000
[2015-06-07 09:01:04.465850] W [gfdb_sqlite3.h:238:gfdb_set_sql_params] 0-vmimage-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal
[2015-06-07 09:01:04.465882] W [gfdb_sqlite3.h:238:gfdb_set_sql_params] 0-vmimage-changetimerecorder: Failed to retrieve sql-db-wal-autocheckpoint from params.Assigning default value: 1000
[2015-06-07 09:01:04.465899] W [gfdb_sqlite3.h:238:gfdb_set_sql_params] 0-vmimage-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: normal
[2015-06-07 09:01:04.465914] W [gfdb_sqlite3.h:238:gfdb_set_sql_params] 0-vmimage-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none
[2015-06-07 09:01:04.479788] I [trash.c:2363:init] 0-vmimage-trash: no option specified for 'eliminate', using NULL
[2015-06-07 09:01:04.593752] W [posix.c:5957:init] 0-vmimage-posix: Posix access control list is not supported.
[2015-06-07 09:01:04.610103] W [graph.c:357:_log_if_unknown_option] 0-vmimage-server: option 'rpc-auth.auth-glusterfs' is not recognized
[2015-06-07 09:01:04.610179] W [graph.c:357:_log_if_unknown_option] 0-vmimage-server: option 'rpc-auth.auth-unix' is not recognized
[2015-06-07 09:01:04.610221] W [graph.c:357:_log_if_unknown_option] 0-vmimage-server: option 'rpc-auth.auth-null' is not recognized
[2015-06-07 09:01:04.610297] W [graph.c:357:_log_if_unknown_option] 0-vmimage-quota: option 'timeout' is not recognized
[2015-06-07 09:01:04.610369] W [graph.c:357:_log_if_unknown_option] 0-vmimage-trash: option 'brick-path' is not recognized
[2015-06-07 09:01:04.660877] W [posix.c:1341:posix_mkdir] 0-vmimage-posix: mkdir (/.trashcan/): gfid (00000000-0000-0000-0000-000000000005) is already associated with directory (/export/gluster01/brick/.glusterfs/00/00/00000000-0000-0000-0000-000000000001/.trashcan). Hence, both directories will share same gfid and this can lead to inconsistencies.
[2015-06-07 09:01:04.660960] E [posix.c:1364:posix_mkdir] 0-vmimage-posix: mkdir of /export/gluster01/brick/.trashcan/ failed: File exists
[2015-06-07 09:01:04.673897] W [posix.c:1341:posix_mkdir] 0-vmimage-posix: mkdir (/.trashcan/internal_op): gfid (00000000-0000-0000-0000-000000000006) is already associated with directory (/export/gluster01/brick/.glusterfs/00/00/00000000-0000-0000-0000-000000000005/internal_op). Hence, both directories will share same gfid and this can lead to inconsistencies.
[2015-06-07 09:01:04.673960] E [posix.c:1364:posix_mkdir] 0-vmimage-posix: mkdir of /export/gluster01/brick/.trashcan/internal_op failed: File exists
[2015-06-07 09:01:04.673984] E [trash.c:387:trash_internal_op_mkdir_cbk] 0-vmimage-trash: mkdir failed for internal op directory : File exists
Final graph:
+------------------------------------------------------------------------------+
1: volume vmimage-posix
2: type storage/posix
3: option glusterd-uuid 26186ec6-a8c7-4834-bcaa-24e30289dba3
4: option directory /export/gluster01/brick
5: option volume-id a8f6ec62-bd8f-4734-af73-6876b2f09488
6: option brick-uid 36
7: option brick-gid 36
8: end-volume
9:
10: volume vmimage-trash
11: type features/trash
12: option trash-dir .trashcan
13: option brick-path /export/gluster01/brick
14: option trash-internal-op off
15: subvolumes vmimage-posix
16: end-volume
17:
18: volume vmimage-changetimerecorder
19: type features/changetimerecorder
20: option db-type sqlite3
21: option hot-brick off
22: option db-name brick.db
23: option db-path /export/gluster01/brick/.glusterfs/
24: option record-exit off
25: option ctr_link_consistency off
26: option record-entry on
27: option ctr-enabled off
28: option record-counters off
29: subvolumes vmimage-trash
30: end-volume
31:
32: volume vmimage-changelog
33: type features/changelog
34: option changelog-brick /export/gluster01/brick
35: option changelog-dir /export/gluster01/brick/.glusterfs/changelogs
36: option changelog-barrier-timeout 120
37: subvolumes vmimage-changetimerecorder
38: end-volume
39:
40: volume vmimage-bitrot-stub
41: type features/bitrot-stub
42: option export /export/gluster01/brick
43: subvolumes vmimage-changelog
44: end-volume
45:
46: volume vmimage-access-control
47: type features/access-control
48: subvolumes vmimage-bitrot-stub
49: end-volume
50:
51: volume vmimage-locks
52: type features/locks
53: subvolumes vmimage-access-control
54: end-volume
55:
56: volume vmimage-upcall
57: type features/upcall
58: option cache-invalidation off
59: subvolumes vmimage-locks
60: end-volume
61:
62: volume vmimage-io-threads
63: type performance/io-threads
64: subvolumes vmimage-upcall
65: end-volume
66:
67: volume vmimage-barrier
68: type features/barrier
69: option barrier disable
70: option barrier-timeout 120
71: subvolumes vmimage-io-threads
72: end-volume
73:
74: volume vmimage-index
75: type features/index
76: option index-base /export/gluster01/brick/.glusterfs/indices
77: subvolumes vmimage-barrier
78: end-volume
79:
80: volume vmimage-marker
81: type features/marker
82: option volume-uuid a8f6ec62-bd8f-4734-af73-6876b2f09488
83: option timestamp-file /var/lib/glusterd/vols/vmimage/marker.tstamp
84: option xtime off
85: option gsync-force-xtime off
86: option quota off
87: option inode-quota off
88: subvolumes vmimage-index
89: end-volume
90:
91: volume vmimage-quota
92: type features/quota
93: option volume-uuid vmimage
94: option server-quota off
95: option timeout 0
96: option deem-statfs off
97: subvolumes vmimage-marker
98: end-volume
99:
100: volume vmimage-worm
101: type features/worm
102: option worm off
103: subvolumes vmimage-quota
104: end-volume
105:
106: volume vmimage-read-only
107: type features/read-only
108: option read-only off
109: subvolumes vmimage-worm
110: end-volume
111:
112: volume /export/gluster01/brick
113: type debug/io-stats
114: option latency-measurement off
115: option count-fop-hits off
116: subvolumes vmimage-read-only
117: end-volume
118:
119: volume vmimage-server
120: type protocol/server
121: option transport.socket.listen-port 49153
122: option rpc-auth.auth-glusterfs on
123: option rpc-auth.auth-unix on
124: option rpc-auth.auth-null on
125: option transport-type tcp
126: option auth.login./export/gluster01/brick.allow 86c3b46e-4b50-4e11-9ef4-cb8395a82c7d
127: option auth.login.86c3b46e-4b50-4e11-9ef4-cb8395a82c7d.password 81ef4a15-1376-4a39-b0e1-4dba3ac698a3
128: option auth.addr./export/gluster01/brick.allow *
129: option rpc-auth-allow-insecure on
130: subvolumes /export/gluster01/brick
131: end-volume
132:
+------------------------------------------------------------------------------+
[2015-06-07 09:01:04.792041] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 86c3b46e-4b50-4e11-9ef4-cb8395a82c7d
[2015-06-07 09:01:04.792101] I [server-handshake.c:585:server_setvolume] 0-vmimage-server: accepted client from ankh.int.rdmedia.com-5321-2015/05/21-03:54:34:352657-vmimage-client-0-0-1 (version: 3.7.0)
[2015-06-07 09:01:04.792796] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 86c3b46e-4b50-4e11-9ef4-cb8395a82c7d
[2015-06-07 09:01:04.792840] I [server-handshake.c:585:server_setvolume] 0-vmimage-server: accepted client from morpork.int.rdmedia.com-3463-2015/06/07-09:01:04:185828-vmimage-client-0-0-0 (version: 3.7.0)
[2015-06-07 09:01:04.832566] I [server-handshake.c:585:server_setvolume] 0-vmimage-server: accepted client from dibbler.int.rdmedia.com-25881-2015/05/21-04:29:49:844757-vmimage-client-0-0-2 (version: 3.7.0)
[2015-06-07 09:01:04.999564] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 86c3b46e-4b50-4e11-9ef4-cb8395a82c7d
[2015-06-07 09:01:04.999626] I [server-handshake.c:585:server_setvolume] 0-vmimage-server: accepted client from morpork.int.rdmedia.com-3455-2015/06/07-09:01:03:151816-vmimage-client-0-0-0 (version: 3.7.0)
[2015-06-07 09:01:05.383314] I [server-handshake.c:585:server_setvolume] 0-vmimage-server: accepted client from dorfl.int.rdmedia.com-31045-2015/05/21-04:52:48:477810-vmimage-client-0-0-1 (version: 3.7.0)
[2015-06-07 09:01:05.448306] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 86c3b46e-4b50-4e11-9ef4-cb8395a82c7d
[2015-06-07 09:01:05.448355] I [server-handshake.c:585:server_setvolume] 0-vmimage-server: accepted client from ankh.int.rdmedia.com-5313-2015/05/21-03:54:33:267820-vmimage-client-0-0-1 (version: 3.7.0)
[2015-06-07 01:46:02.230568] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2015-06-07 08:59:30.515512] W [socket.c:642:__socket_rwv] 0-conf-client-0: readv on 10.100.3.10:49159 failed (No data available) [2015-06-07 08:59:30.530767] W [socket.c:642:__socket_rwv] 0-vmimage-client-0: readv on 10.100.3.10:49153 failed (No data available) [2015-06-07 08:59:30.553980] I [client.c:2086:client_rpc_notify] 0-conf-client-0: disconnected from conf-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:30.568315] W [socket.c:642:__socket_rwv] 0-mssqlp-client-0: readv on 10.100.3.10:49163 failed (No data available) [2015-06-07 08:59:30.568366] I [client.c:2086:client_rpc_notify] 0-mssqlp-client-0: disconnected from mssqlp-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:30.568483] W [socket.c:642:__socket_rwv] 0-fl-iiscfg-client-1: readv on 10.100.3.10:49164 failed (No data available) [2015-06-07 08:59:30.568520] I [client.c:2086:client_rpc_notify] 0-fl-iiscfg-client-1: disconnected from fl-iiscfg-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:30.568574] W [socket.c:642:__socket_rwv] 0-IT-client-1: readv on 10.100.3.10:49171 failed (No data available) [2015-06-07 08:59:30.568590] I [client.c:2086:client_rpc_notify] 0-IT-client-1: disconnected from IT-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:30.568635] W [socket.c:642:__socket_rwv] 0-rb-appcfg-client-0: readv on 10.100.3.10:49169 failed (No data available) [2015-06-07 08:59:30.568651] I [client.c:2086:client_rpc_notify] 0-rb-appcfg-client-0: disconnected from rb-appcfg-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:30.568684] W [socket.c:642:__socket_rwv] 0-rb-iiscfg-client-1: readv on 10.100.3.10:49165 failed (No data available) [2015-06-07 08:59:30.568700] I [client.c:2086:client_rpc_notify] 0-rb-iiscfg-client-1: disconnected from rb-iiscfg-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:30.568731] W [socket.c:642:__socket_rwv] 0-wwwdata-client-0: readv on 10.100.3.10:49172 failed (No data available) [2015-06-07 08:59:30.568746] I [client.c:2086:client_rpc_notify] 0-wwwdata-client-0: disconnected from wwwdata-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:30.568776] W [socket.c:642:__socket_rwv] 0-ovirt-export-client-0: readv on 10.100.3.10:49170 failed (No data available) [2015-06-07 08:59:30.568791] I [client.c:2086:client_rpc_notify] 0-ovirt-export-client-0: disconnected from ovirt-export-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:30.574272] W [socket.c:642:__socket_rwv] 0-lpx-assets-client-0: readv on 10.100.3.10:49173 failed (No data available) [2015-06-07 08:59:30.574313] I [client.c:2086:client_rpc_notify] 0-lpx-assets-client-0: disconnected from lpx-assets-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:30.576293] W [socket.c:642:__socket_rwv] 0-fl-webroot-client-1: readv on 10.100.3.10:49167 failed (No data available) [2015-06-07 08:59:30.576334] I [client.c:2086:client_rpc_notify] 0-fl-webroot-client-1: disconnected from fl-webroot-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:30.583946] W [socket.c:642:__socket_rwv] 0-rb-webroot-client-1: readv on 10.100.3.10:49166 failed (No data available) [2015-06-07 08:59:30.584011] I [client.c:2086:client_rpc_notify] 0-rb-webroot-client-1: disconnected from rb-webroot-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:30.620511] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f0517023fb0] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7f0516df4a87] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f0516df4b9e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7f0516df4c6b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7f0516df522f] ))))) 0-vmimage-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2015-06-07 08:59:30.508489 (xid=0x8107f) [2015-06-07 08:59:30.620576] E [client-rpc-fops.c:1528:client3_3_inodelk_cbk] 0-vmimage-client-0: remote operation failed: Transport endpoint is not connected [2015-06-07 08:59:30.620614] I [client.c:2086:client_rpc_notify] 0-vmimage-client-0: disconnected from vmimage-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:30.621125] W [client-rpc-fops.c:5639:client3_3_rchecksum] 0-vmimage-client-0: (950a47c7-5d0e-457f-a6da-61d2c2ba0e1c) remote_fd is -1. EBADFD [2015-06-07 08:59:30.656535] I [socket.c:3358:socket_submit_request] 0-vmimage-client-0: not connected (priv->connected = 0) [2015-06-07 08:59:30.656602] W [rpc-clnt.c:1566:rpc_clnt_submit] 0-vmimage-client-0: failed to submit rpc-request (XID: 0x81080 Program: GlusterFS 3.3, ProgVers: 330, Proc: 13) to rpc-transport (vmimage-client-0) [2015-06-07 08:59:30.656640] W [client-rpc-fops.c:853:client3_3_writev_cbk] 0-vmimage-client-0: remote operation failed: Transport endpoint is not connected [2015-06-07 08:59:30.656683] W [client-rpc-fops.c:4247:client3_3_writev] 0-vmimage-client-0: failed to send the fop [2015-06-07 08:59:30.656964] E [client-handshake.c:1488:client_query_portmap_cbk] 0-vmimage-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-06-07 08:59:30.657054] I [client.c:2086:client_rpc_notify] 0-vmimage-client-0: disconnected from vmimage-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:30.657365] E [client-rpc-fops.c:1528:client3_3_inodelk_cbk] 0-vmimage-client-0: remote operation failed: Transport endpoint is not connected [2015-06-07 08:59:30.657707] E [client-rpc-fops.c:1528:client3_3_inodelk_cbk] 0-vmimage-client-0: remote operation failed: Transport endpoint is not connected [2015-06-07 08:59:30.669164] W [client-rpc-fops.c:1092:client3_3_getxattr_cbk] 0-vmimage-client-0: remote operation failed: Transport endpoint is not connected. Path: <gfid:f90e8747-ca3e-42fa-940f-3d82353c596b> (f90e8747-ca3e-42fa-940f-3d82353c596b). Key: glusterfs.gfid2path [2015-06-07 08:59:30.669236] W [client-rpc-fops.c:1092:client3_3_getxattr_cbk] 0-vmimage-client-0: remote operation failed: Transport endpoint is not connected. Path: <gfid:942a9620-90ce-48d6-9ea5-5a0972dc2e7e> (942a9620-90ce-48d6-9ea5-5a0972dc2e7e). Key: glusterfs.gfid2path [2015-06-07 08:59:30.669281] W [client-rpc-fops.c:1092:client3_3_getxattr_cbk] 0-vmimage-client-0: remote operation failed: Transport endpoint is not connected. Path: <gfid:8890e437-fb6d-4704-a6d1-899e9bb55ee9> (8890e437-fb6d-4704-a6d1-899e9bb55ee9). Key: glusterfs.gfid2path [2015-06-07 08:59:30.669319] W [client-rpc-fops.c:1092:client3_3_getxattr_cbk] 0-vmimage-client-0: remote operation failed: Transport endpoint is not connected. Path: <gfid:3186cd00-e203-4bc5-890e-dc026d60a61c> (3186cd00-e203-4bc5-890e-dc026d60a61c). Key: glusterfs.gfid2path [2015-06-07 08:59:30.669355] W [client-rpc-fops.c:1092:client3_3_getxattr_cbk] 0-vmimage-client-0: remote operation failed: Transport endpoint is not connected. Path: <gfid:f9ae95bb-e68b-49bd-95e5-11aa3bac4e81> (f9ae95bb-e68b-49bd-95e5-11aa3bac4e81). Key: glusterfs.gfid2path [2015-06-07 08:59:30.669391] W [client-rpc-fops.c:1092:client3_3_getxattr_cbk] 0-vmimage-client-0: remote operation failed: Transport endpoint is not connected. Path: <gfid:5ebf44d1-17c9-47a5-901d-780f31e8a0f8> (5ebf44d1-17c9-47a5-901d-780f31e8a0f8). Key: glusterfs.gfid2path [2015-06-07 08:59:30.669432] W [client-rpc-fops.c:1092:client3_3_getxattr_cbk] 0-vmimage-client-0: remote operation failed: Transport endpoint is not connected. Path: <gfid:1e2d6c85-cf83-4b19-bd89-f2d7c6d9fe8b> (1e2d6c85-cf83-4b19-bd89-f2d7c6d9fe8b). Key: glusterfs.gfid2path [2015-06-07 08:59:30.669488] W [client-rpc-fops.c:1092:client3_3_getxattr_cbk] 0-vmimage-client-0: remote operation failed: Transport endpoint is not connected. Path: <gfid:d8919fa2-10fb-4950-9607-b15027bdd40f> (d8919fa2-10fb-4950-9607-b15027bdd40f). Key: glusterfs.gfid2path [2015-06-07 08:59:30.669526] W [client-rpc-fops.c:1092:client3_3_getxattr_cbk] 0-vmimage-client-0: remote operation failed: Transport endpoint is not connected. Path: <gfid:07ab210e-53c2-4bbb-90ac-ff7407f5294a> (07ab210e-53c2-4bbb-90ac-ff7407f5294a). Key: glusterfs.gfid2path [2015-06-07 08:59:30.669560] W [client-rpc-fops.c:5696:client3_3_readdir] 0-vmimage-client-0: (d77fcc82-7319-4d75-80e5-a9a1b9092d6c) remote_fd is -1. EBADFD [2015-06-07 08:59:41.050817] E [client-handshake.c:1488:client_query_portmap_cbk] 0-mssqlp-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-06-07 08:59:41.051016] I [client.c:2086:client_rpc_notify] 0-mssqlp-client-0: disconnected from mssqlp-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:41.051190] E [client-handshake.c:1488:client_query_portmap_cbk] 0-fl-iiscfg-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-06-07 08:59:41.051358] E [client-handshake.c:1488:client_query_portmap_cbk] 0-IT-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-06-07 08:59:41.051470] I [client.c:2086:client_rpc_notify] 0-fl-iiscfg-client-1: disconnected from fl-iiscfg-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:41.051593] E [client-handshake.c:1488:client_query_portmap_cbk] 0-rb-appcfg-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-06-07 08:59:41.051648] I [client.c:2086:client_rpc_notify] 0-IT-client-1: disconnected from IT-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:41.051849] E [client-handshake.c:1488:client_query_portmap_cbk] 0-rb-iiscfg-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-06-07 08:59:41.051903] I [client.c:2086:client_rpc_notify] 0-rb-appcfg-client-0: disconnected from rb-appcfg-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:41.052051] E [client-handshake.c:1488:client_query_portmap_cbk] 0-wwwdata-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-06-07 08:59:41.052148] I [client.c:2086:client_rpc_notify] 0-rb-iiscfg-client-1: disconnected from rb-iiscfg-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:41.052298] E [client-handshake.c:1488:client_query_portmap_cbk] 0-ovirt-export-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-06-07 08:59:41.052409] I [client.c:2086:client_rpc_notify] 0-wwwdata-client-0: disconnected from wwwdata-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:41.052458] E [client-handshake.c:1488:client_query_portmap_cbk] 0-lpx-assets-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-06-07 08:59:41.052610] E [client-handshake.c:1488:client_query_portmap_cbk] 0-fl-webroot-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-06-07 08:59:41.052678] I [client.c:2086:client_rpc_notify] 0-ovirt-export-client-0: disconnected from ovirt-export-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:41.052822] I [client.c:2086:client_rpc_notify] 0-lpx-assets-client-0: disconnected from lpx-assets-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:41.052838] E [client-handshake.c:1488:client_query_portmap_cbk] 0-rb-webroot-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-06-07 08:59:41.052901] I [client.c:2086:client_rpc_notify] 0-fl-webroot-client-1: disconnected from fl-webroot-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:41.052948] E [client-handshake.c:1488:client_query_portmap_cbk] 0-conf-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running. [2015-06-07 08:59:41.053017] I [client.c:2086:client_rpc_notify] 0-rb-webroot-client-1: disconnected from rb-webroot-client-1. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 08:59:41.053038] I [client.c:2086:client_rpc_notify] 0-conf-client-0: disconnected from conf-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-06-07 09:00:51.908950] W [socket.c:642:__socket_rwv] 0-glusterfs: readv on 127.0.0.1:24007 failed (No data available) [2015-06-07 09:01:03.179809] W [glusterfsd.c:1219:cleanup_and_exit] (--> 0-: received signum (15), shutting down [2015-06-07 09:01:04.194300] I [MSGID: 100030] [glusterfsd.c:2294:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.0 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c0d7454986c96eef463d028dc8bce9fe.socket --xlator-option *replicate*.node-uuid=26186ec6-a8c7-4834-bcaa-24e30289dba3) [2015-06-07 09:01:04.205771] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-06-07 09:01:04.396639] I [graph.c:269:gf_add_cmdline_options] 0-IT-replicate-0: adding option 'node-uuid' for volume 'IT-replicate-0' with value '26186ec6-a8c7-4834-bcaa-24e30289dba3' [2015-06-07 09:01:04.396713] I [graph.c:269:gf_add_cmdline_options] 0-conf-replicate-0: adding option 'node-uuid' for volume 'conf-replicate-0' with value '26186ec6-a8c7-4834-bcaa-24e30289dba3' [2015-06-07 09:01:04.396754] I [graph.c:269:gf_add_cmdline_options] 0-fl-iiscfg-replicate-0: adding option 'node-uuid' for volume 'fl-iiscfg-replicate-0' with value '26186ec6-a8c7-4834-bcaa-24e30289dba3' [2015-06-07 09:01:04.396791] I [graph.c:269:gf_add_cmdline_options] 0-fl-webroot-replicate-0: adding option 'node-uuid' for volume 'fl-webroot-replicate-0' with value '26186ec6-a8c7-4834-bcaa-24e30289dba3' [2015-06-07 09:01:04.396828] I [graph.c:269:gf_add_cmdline_options] 0-lpx-assets-replicate-0: adding option 'node-uuid' for volume 'lpx-assets-replicate-0' with value '26186ec6-a8c7-4834-bcaa-24e30289dba3' [2015-06-07 09:01:04.396862] I [graph.c:269:gf_add_cmdline_options] 0-mssqlp-replicate-0: adding option 'node-uuid' for volume 'mssqlp-replicate-0' with value '26186ec6-a8c7-4834-bcaa-24e30289dba3' [2015-06-07 09:01:04.396895] I [graph.c:269:gf_add_cmdline_options] 0-ovirt-export-replicate-0: adding option 'node-uuid' for volume 'ovirt-export-replicate-0' with value '26186ec6-a8c7-4834-bcaa-24e30289dba3' [2015-06-07 09:01:04.396928] I [graph.c:269:gf_add_cmdline_options] 0-rb-appcfg-replicate-0: adding option 'node-uuid' for volume 'rb-appcfg-replicate-0' with value '26186ec6-a8c7-4834-bcaa-24e30289dba3' [2015-06-07 09:01:04.396970] I [graph.c:269:gf_add_cmdline_options] 0-rb-iiscfg-replicate-0: adding option 'node-uuid' for volume 'rb-iiscfg-replicate-0' with value '26186ec6-a8c7-4834-bcaa-24e30289dba3' [2015-06-07 09:01:04.396996] I [graph.c:269:gf_add_cmdline_options] 0-rb-webroot-replicate-0: adding option 'node-uuid' for volume 'rb-webroot-replicate-0' with value '26186ec6-a8c7-4834-bcaa-24e30289dba3' [2015-06-07 09:01:04.397038] I [graph.c:269:gf_add_cmdline_options] 0-vmimage-replicate-0: adding option 'node-uuid' for volume 'vmimage-replicate-0' with value '26186ec6-a8c7-4834-bcaa-24e30289dba3' [2015-06-07 09:01:04.397066] I [graph.c:269:gf_add_cmdline_options] 0-wwwdata-replicate-0: adding option 'node-uuid' for volume 'wwwdata-replicate-0' with value '26186ec6-a8c7-4834-bcaa-24e30289dba3' [2015-06-07 09:01:04.403625] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2015-06-07 09:01:04.471531] I [client.c:2154:notify] 0-IT-client-0: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.471963] I [client.c:2154:notify] 0-IT-client-1: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.472383] I [client.c:2154:notify] 0-conf-client-0: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.472757] I [client.c:2154:notify] 0-conf-client-1: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.473046] I [client.c:2154:notify] 0-fl-iiscfg-client-0: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.473382] I [client.c:2154:notify] 0-fl-iiscfg-client-1: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.473751] I [client.c:2154:notify] 0-fl-webroot-client-0: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.474039] I [client.c:2154:notify] 0-fl-webroot-client-1: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.474383] I [client.c:2154:notify] 0-lpx-assets-client-0: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.474812] I [client.c:2154:notify] 0-lpx-assets-client-1: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.475365] I [client.c:2154:notify] 0-mssqlp-client-0: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.475790] I [client.c:2154:notify] 0-mssqlp-client-1: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.476098] I [client.c:2154:notify] 0-ovirt-export-client-0: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.476512] I [client.c:2154:notify] 0-ovirt-export-client-1: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.476829] I [client.c:2154:notify] 0-rb-appcfg-client-0: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.477207] I [client.c:2154:notify] 0-rb-appcfg-client-1: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.477527] I [client.c:2154:notify] 0-rb-iiscfg-client-0: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.477908] I [client.c:2154:notify] 0-rb-iiscfg-client-1: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.478328] I [client.c:2154:notify] 0-rb-webroot-client-0: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.478665] I [client.c:2154:notify] 0-rb-webroot-client-1: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.479036] I [client.c:2154:notify] 0-vmimage-client-0: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.479372] I [client.c:2154:notify] 0-vmimage-client-1: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.479699] I [client.c:2154:notify] 0-wwwdata-client-0: parent translators are ready, attempting connect on transport [2015-06-07 09:01:04.480108] I [client.c:2154:notify] 0-wwwdata-client-1: parent translators are ready, attempting connect on transport Final graph: +------------------------------------------------------------------------------+ 1: volume IT-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host 10.100.3.11 5: option remote-subvolume /export/glu/common/IT 6: option transport-type socket 7: option username 036ea1e2-08dc-4adb-9239-e6df923ec8bf 8: option password bd654da2-3b8e-4fc6-a6a8-49d688fade36 9: end-volume 10: 11: volume IT-client-1 12: type protocol/client 13: option ping-timeout 42 14: option remote-host 10.100.3.10 15: option remote-subvolume /export/glu/common/IT 16: option transport-type socket 17: option username 036ea1e2-08dc-4adb-9239-e6df923ec8bf 18: option password bd654da2-3b8e-4fc6-a6a8-49d688fade36 19: end-volume 20: 21: volume IT-replicate-0 22: type cluster/replicate 23: option node-uuid 26186ec6-a8c7-4834-bcaa-24e30289dba3 24: option background-self-heal-count 0 25: option metadata-self-heal on 26: option data-self-heal on 27: option entry-self-heal on 28: option self-heal-daemon enable 29: option iam-self-heal-daemon yes 30: subvolumes IT-client-0 IT-client-1 31: end-volume 32: 33: volume conf-client-0 34: type protocol/client 35: option ping-timeout 42 36: option remote-host 10.100.3.10 37: option remote-subvolume /export/glu/conf 38: option transport-type socket 39: option username 6d5b2bb4-81da-4b0b-b756-228678397eaa 40: option password 35976b1e-fce6-47c5-b483-aa6a91844cf4 41: end-volume 42: 43: volume conf-client-1 44: type protocol/client 45: option ping-timeout 42 46: option remote-host 10.100.3.11 47: option remote-subvolume /export/glu/conf 48: option transport-type socket 49: option username 6d5b2bb4-81da-4b0b-b756-228678397eaa 50: option password 35976b1e-fce6-47c5-b483-aa6a91844cf4 51: end-volume 52: 53: volume conf-replicate-0 54: type cluster/replicate 55: option node-uuid 26186ec6-a8c7-4834-bcaa-24e30289dba3 56: option background-self-heal-count 0 57: option metadata-self-heal on 58: option data-self-heal on 59: option entry-self-heal on 60: option self-heal-daemon enable 61: option iam-self-heal-daemon yes 62: subvolumes conf-client-0 conf-client-1 63: end-volume 64: 65: volume fl-iiscfg-client-0 66: type protocol/client 67: option ping-timeout 42 68: option remote-host 10.100.3.11 69: option remote-subvolume /export/glu/web/flash/iiscfg 70: option transport-type socket 71: option username 64cb0ed6-fc18-4ace-9ab2-a79fa8590ee9 72: option password 095f82ab-8d21-4008-8858-f589999aa78b 73: end-volume 74: 75: volume fl-iiscfg-client-1 76: type protocol/client 77: option ping-timeout 42 78: option remote-host 10.100.3.10 79: option remote-subvolume /export/glu/web/flash/iiscfg 80: option transport-type socket 81: option username 64cb0ed6-fc18-4ace-9ab2-a79fa8590ee9 82: option password 095f82ab-8d21-4008-8858-f589999aa78b 83: end-volume 84: 85: volume fl-iiscfg-replicate-0 86: type cluster/replicate 87: option node-uuid 26186ec6-a8c7-4834-bcaa-24e30289dba3 88: option background-self-heal-count 0 89: option metadata-self-heal on 90: option data-self-heal on 91: option entry-self-heal on 92: option self-heal-daemon enable 93: option iam-self-heal-daemon yes 94: subvolumes fl-iiscfg-client-0 fl-iiscfg-client-1 95: end-volume 96: 97: volume fl-webroot-client-0 98: type protocol/client 99: option ping-timeout 42 100: option remote-host 10.100.3.11 101: option remote-subvolume /export/glu/web/flash/webroot 102: option transport-type socket 103: option username 7a8a4cd4-1948-460b-abd8-b1734ea9ccbb 104: option password cb389c62-e8e0-47bd-8e87-bb4a2fd0667d 105: end-volume 106: 107: volume fl-webroot-client-1 108: type protocol/client 109: option ping-timeout 42 110: option remote-host 10.100.3.10 111: option remote-subvolume /export/glu/web/flash/webroot 112: option transport-type socket 113: option username 7a8a4cd4-1948-460b-abd8-b1734ea9ccbb 114: option password cb389c62-e8e0-47bd-8e87-bb4a2fd0667d 115: end-volume 116: 117: volume fl-webroot-replicate-0 118: type cluster/replicate 119: option node-uuid 26186ec6-a8c7-4834-bcaa-24e30289dba3 120: option background-self-heal-count 0 121: option metadata-self-heal on 122: option data-self-heal on 123: option entry-self-heal on 124: option self-heal-daemon enable 125: option iam-self-heal-daemon yes 126: subvolumes fl-webroot-client-0 fl-webroot-client-1 127: end-volume 128: 129: volume lpx-assets-client-0 130: type protocol/client 131: option ping-timeout 42 132: option remote-host 10.100.3.10 133: option remote-subvolume /export/glu/web/lpx 134: option transport-type socket 135: option username e0cb16b2-066c-4fcb-930d-bc546f93b165 136: option password 12301fa4-265d-4a6b-81be-64da7a7ae7a8 137: end-volume 138: 139: volume lpx-assets-client-1 140: type protocol/client 141: option ping-timeout 42 142: option remote-host 10.100.3.11 143: option remote-subvolume /export/glu/web/lpx 144: option transport-type socket 145: option username e0cb16b2-066c-4fcb-930d-bc546f93b165 146: option password 12301fa4-265d-4a6b-81be-64da7a7ae7a8 147: end-volume 148: 149: volume lpx-assets-replicate-0 150: type cluster/replicate 151: option node-uuid 26186ec6-a8c7-4834-bcaa-24e30289dba3 152: option background-self-heal-count 0 153: option metadata-self-heal on 154: option data-self-heal on 155: option entry-self-heal on 156: option self-heal-daemon enable 157: option iam-self-heal-daemon yes 158: subvolumes lpx-assets-client-0 lpx-assets-client-1 159: end-volume 160: 161: volume mssqlp-client-0 162: type protocol/client 163: option ping-timeout 42 164: option remote-host 10.100.3.10 165: option remote-subvolume /export/glu/db/mssql/p 166: option transport-type socket 167: option username 0139ba8a-5313-4773-a98a-322264b4775e 168: option password 9c4c45c6-014d-4fd0-b753-8f0cf9a1782c 169: end-volume 170: 171: volume mssqlp-client-1 172: type protocol/client 173: option ping-timeout 42 174: option remote-host 10.100.3.11 175: option remote-subvolume /export/glu/db/mssql/p 176: option transport-type socket 177: option username 0139ba8a-5313-4773-a98a-322264b4775e 178: option password 9c4c45c6-014d-4fd0-b753-8f0cf9a1782c 179: end-volume 180: 181: volume mssqlp-replicate-0 182: type cluster/replicate 183: option node-uuid 26186ec6-a8c7-4834-bcaa-24e30289dba3 184: option background-self-heal-count 0 185: option metadata-self-heal on 186: option data-self-heal on 187: option entry-self-heal on 188: option self-heal-daemon enable 189: option iam-self-heal-daemon yes 190: subvolumes mssqlp-client-0 mssqlp-client-1 191: end-volume 192: 193: volume ovirt-export-client-0 194: type protocol/client 195: option ping-timeout 42 196: option remote-host 10.100.3.10 197: option remote-subvolume /export/glu/vm/ovirt-export 198: option transport-type socket 199: option username 2fae2272-81cb-4627-8db8-65f09290aa31 200: option password 524fe375-5fe4-4159-b765-a2e4d9a11d51 201: option filter-O_DIRECT on 202: end-volume 203: 204: volume ovirt-export-client-1 205: type protocol/client 206: option ping-timeout 42 207: option remote-host 10.100.3.11 208: option remote-subvolume /export/glu/vm/ovirt-export 209: option transport-type socket 210: option username 2fae2272-81cb-4627-8db8-65f09290aa31 211: option password 524fe375-5fe4-4159-b765-a2e4d9a11d51 212: option filter-O_DIRECT on 213: end-volume 214: 215: volume ovirt-export-replicate-0 216: type cluster/replicate 217: option node-uuid 26186ec6-a8c7-4834-bcaa-24e30289dba3 218: option background-self-heal-count 0 219: option metadata-self-heal on 220: option data-self-heal on 221: option entry-self-heal on 222: option self-heal-daemon enable 223: option eager-lock enable 224: option iam-self-heal-daemon yes 225: subvolumes ovirt-export-client-0 ovirt-export-client-1 226: end-volume 227: 228: volume rb-appcfg-client-0 229: type protocol/client 230: option ping-timeout 42 231: option remote-host 10.100.3.10 232: option remote-subvolume /export/glu/web/rebuild/appcfg 233: option transport-type socket 234: option username 9b0d5192-598a-408b-86d2-ca3a553189fa 235: option password 44d1b767-d689-4280-998f-849325de0543 236: end-volume 237: 238: volume rb-appcfg-client-1 239: type protocol/client 240: option ping-timeout 42 241: option remote-host 10.100.3.11 242: option remote-subvolume /export/glu/web/rebuild/appcfg 243: option transport-type socket 244: option username 9b0d5192-598a-408b-86d2-ca3a553189fa 245: option password 44d1b767-d689-4280-998f-849325de0543 246: end-volume 247: 248: volume rb-appcfg-replicate-0 249: type cluster/replicate 250: option node-uuid 26186ec6-a8c7-4834-bcaa-24e30289dba3 251: option background-self-heal-count 0 252: option metadata-self-heal on 253: option data-self-heal on 254: option entry-self-heal on 255: option self-heal-daemon enable 256: option iam-self-heal-daemon yes 257: subvolumes rb-appcfg-client-0 rb-appcfg-client-1 258: end-volume 259: 260: volume rb-iiscfg-client-0 261: type protocol/client 262: option ping-timeout 42 263: option remote-host 10.100.3.11 264: option remote-subvolume /export/glu/web/rebuild/iiscfg 265: option transport-type socket 266: option username 7900560e-ac67-4920-a018-46110a7c4af9 267: option password 8eb286f1-dda8-4cc7-8e48-a93e5f1033c6 268: end-volume 269: 270: volume rb-iiscfg-client-1 271: type protocol/client 272: option ping-timeout 42 273: option remote-host 10.100.3.10 274: option remote-subvolume /export/glu/web/rebuild/iiscfg 275: option transport-type socket 276: option username 7900560e-ac67-4920-a018-46110a7c4af9 277: option password 8eb286f1-dda8-4cc7-8e48-a93e5f1033c6 278: end-volume 279: 280: volume rb-iiscfg-replicate-0 281: type cluster/replicate 282: option node-uuid 26186ec6-a8c7-4834-bcaa-24e30289dba3 283: option background-self-heal-count 0 284: option metadata-self-heal on 285: option data-self-heal on 286: option entry-self-heal on 287: option self-heal-daemon enable 288: option iam-self-heal-daemon yes 289: subvolumes rb-iiscfg-client-0 rb-iiscfg-client-1 290: end-volume 291: 292: volume rb-webroot-client-0 293: type protocol/client 294: option ping-timeout 42 295: option remote-host 10.100.3.11 296: option remote-subvolume /export/glu/web/rebuild/webroot 297: option transport-type socket 298: option username 92ef5f56-15c8-4521-abe8-048a5fc6760f 299: option password 61d55fb2-5082-4ce0-ba3c-40e3198e2f58 300: end-volume 301: 302: volume rb-webroot-client-1 303: type protocol/client 304: option ping-timeout 42 305: option remote-host 10.100.3.10 306: option remote-subvolume /export/glu/web/rebuild/webroot 307: option transport-type socket 308: option username 92ef5f56-15c8-4521-abe8-048a5fc6760f 309: option password 61d55fb2-5082-4ce0-ba3c-40e3198e2f58 310: end-volume 311: 312: volume rb-webroot-replicate-0 313: type cluster/replicate 314: option node-uuid 26186ec6-a8c7-4834-bcaa-24e30289dba3 315: option background-self-heal-count 0 316: option metadata-self-heal on 317: option data-self-heal on 318: option entry-self-heal on 319: option self-heal-daemon enable 320: option iam-self-heal-daemon yes 321: subvolumes rb-webroot-client-0 rb-webroot-client-1 322: end-volume 323: 324: volume vmimage-client-0 325: type protocol/client 326: option ping-timeout 42 327: option remote-host 10.100.3.10 328: option remote-subvolume /export/gluster01/brick 329: option transport-type socket 330: option username 86c3b46e-4b50-4e11-9ef4-cb8395a82c7d 331: option password 81ef4a15-1376-4a39-b0e1-4dba3ac698a3 332: option filter-O_DIRECT on 333: end-volume 334: 335: volume vmimage-client-1 336: type protocol/client 337: option ping-timeout 42 338: option remote-host 10.100.3.11 339: option remote-subvolume /export/gluster01/brick 340: option transport-type socket 341: option username 86c3b46e-4b50-4e11-9ef4-cb8395a82c7d 342: option password 81ef4a15-1376-4a39-b0e1-4dba3ac698a3 343: option filter-O_DIRECT on 344: end-volume 345: 346: volume vmimage-replicate-0 347: type cluster/replicate 348: option node-uuid 26186ec6-a8c7-4834-bcaa-24e30289dba3 349: option background-self-heal-count 0 350: option metadata-self-heal on 351: option data-self-heal on 352: option entry-self-heal on 353: option self-heal-daemon enable 354: option eager-lock enable 355: option iam-self-heal-daemon yes 356: subvolumes vmimage-client-0 vmimage-client-1 357: end-volume 358: 359: volume wwwdata-client-0 360: type protocol/client 361: option ping-timeout 42 362: option remote-host 10.100.3.10 363: option remote-subvolume /export/glu/web/wwwdata 364: option transport-type socket 365: option username 2170f84e-4c32-4a5b-abca-f0930395f6a8 366: option password dd53d248-912f-4fdb-9e80-5b3e7fe5de60 367: end-volume 368: 369: volume wwwdata-client-1 370: type protocol/client 371: option ping-timeout 42 372: option remote-host 10.100.3.11 373: option remote-subvolume /export/glu/web/wwwdata 374: option transport-type socket 375: option username 2170f84e-4c32-4a5b-abca-f0930395f6a8 376: option password dd53d248-912f-4fdb-9e80-5b3e7fe5de60 377: end-volume 378: 379: volume wwwdata-replicate-0 380: type cluster/replicate 381: option node-uuid 26186ec6-a8c7-4834-bcaa-24e30289dba3 382: option background-self-heal-count 0 383: option metadata-self-heal on 384: option data-self-heal on 385: option entry-self-heal on 386: option self-heal-daemon enable 387: option iam-self-heal-daemon yes 388: subvolumes wwwdata-client-0 wwwdata-client-1 389: end-volume 390: 391: volume glustershd 392: type debug/io-stats 393: subvolumes IT-replicate-0 conf-replicate-0 fl-iiscfg-replicate-0 fl-webroot-replicate-0 lpx-assets-replicate-0 mssqlp-replicate-0 ovirt-export-replicate-0 rb-appcfg-replicate-0 rb-iiscfg-replicate-0 rb-webroot-replicate-0 vmimage-replicate-0 wwwdata-replicate-0 394: end-volume 395: +------------------------------------------------------------------------------+ [2015-06-07 09:01:04.492320] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-IT-client-0: changing port to 49171 (from 0) [2015-06-07 09:01:04.492428] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-conf-client-1: changing port to 49159 (from 0) [2015-06-07 09:01:04.492513] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-fl-iiscfg-client-0: changing port to 49164 (from 0) [2015-06-07 09:01:04.493049] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-fl-webroot-client-0: changing port to 49167 (from 0) [2015-06-07 09:01:04.493699] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-lpx-assets-client-1: changing port to 49173 (from 0) [2015-06-07 09:01:04.493800] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-mssqlp-client-1: changing port to 49163 (from 0) [2015-06-07 09:01:04.493810] I [client-handshake.c:1405:select_server_supported_programs] 0-IT-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.493981] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-rb-appcfg-client-1: changing port to 49169 (from 0) [2015-06-07 09:01:04.493959] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-ovirt-export-client-1: changing port to 49170 (from 0) [2015-06-07 09:01:04.494030] I [client-handshake.c:1405:select_server_supported_programs] 0-conf-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.494287] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-rb-iiscfg-client-0: changing port to 49165 (from 0) [2015-06-07 09:01:04.494774] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-rb-webroot-client-0: changing port to 49166 (from 0) [2015-06-07 09:01:04.495222] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-vmimage-client-1: changing port to 49153 (from 0) [2015-06-07 09:01:04.495381] I [client-handshake.c:1405:select_server_supported_programs] 0-fl-webroot-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.495502] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-wwwdata-client-1: changing port to 49172 (from 0) [2015-06-07 09:01:04.496301] I [client-handshake.c:1405:select_server_supported_programs] 0-mssqlp-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.496364] I [client-handshake.c:1405:select_server_supported_programs] 0-lpx-assets-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.496741] I [client-handshake.c:1405:select_server_supported_programs] 0-wwwdata-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.501573] I [client-handshake.c:1405:select_server_supported_programs] 0-rb-iiscfg-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.502863] I [client-handshake.c:1405:select_server_supported_programs] 0-vmimage-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.506243] I [client-handshake.c:1405:select_server_supported_programs] 0-ovirt-export-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.506690] I [client-handshake.c:1405:select_server_supported_programs] 0-rb-webroot-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.510378] I [client-handshake.c:1405:select_server_supported_programs] 0-fl-iiscfg-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.519080] I [client-handshake.c:1405:select_server_supported_programs] 0-rb-appcfg-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.537444] I [client-handshake.c:1193:client_setvolume_cbk] 0-rb-webroot-client-0: Connected to rb-webroot-client-0, attached to remote volume '/export/glu/web/rebuild/webroot'. [2015-06-07 09:01:04.539017] I [client-handshake.c:1193:client_setvolume_cbk] 0-rb-appcfg-client-1: Connected to rb-appcfg-client-1, attached to remote volume '/export/glu/web/rebuild/appcfg'. [2015-06-07 09:01:04.560486] I [client-handshake.c:1203:client_setvolume_cbk] 0-rb-webroot-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.560533] I [client-handshake.c:1203:client_setvolume_cbk] 0-rb-appcfg-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.560677] I [MSGID: 108005] [afr-common.c:3880:afr_notify] 0-rb-webroot-replicate-0: Subvolume 'rb-webroot-client-0' came back up; going online. [2015-06-07 09:01:04.560798] I [client-handshake.c:1193:client_setvolume_cbk] 0-lpx-assets-client-1: Connected to lpx-assets-client-1, attached to remote volume '/export/glu/web/lpx'. [2015-06-07 09:01:04.560765] I [MSGID: 108005] [afr-common.c:3880:afr_notify] 0-rb-appcfg-replicate-0: Subvolume 'rb-appcfg-client-1' came back up; going online. [2015-06-07 09:01:04.560838] I [client-handshake.c:1203:client_setvolume_cbk] 0-lpx-assets-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.561010] I [client-handshake.c:1193:client_setvolume_cbk] 0-rb-iiscfg-client-0: Connected to rb-iiscfg-client-0, attached to remote volume '/export/glu/web/rebuild/iiscfg'. [2015-06-07 09:01:04.561068] I [MSGID: 108005] [afr-common.c:3880:afr_notify] 0-lpx-assets-replicate-0: Subvolume 'lpx-assets-client-1' came back up; going online. [2015-06-07 09:01:04.561094] I [client-handshake.c:1203:client_setvolume_cbk] 0-rb-iiscfg-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.561166] I [client-handshake.c:1193:client_setvolume_cbk] 0-mssqlp-client-1: Connected to mssqlp-client-1, attached to remote volume '/export/glu/db/mssql/p'. [2015-06-07 09:01:04.561204] I [client-handshake.c:1203:client_setvolume_cbk] 0-mssqlp-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.561255] I [MSGID: 108005] [afr-common.c:3880:afr_notify] 0-rb-iiscfg-replicate-0: Subvolume 'rb-iiscfg-client-0' came back up; going online. [2015-06-07 09:01:04.561314] I [MSGID: 108005] [afr-common.c:3880:afr_notify] 0-mssqlp-replicate-0: Subvolume 'mssqlp-client-1' came back up; going online. [2015-06-07 09:01:04.561407] I [client-handshake.c:1193:client_setvolume_cbk] 0-fl-iiscfg-client-0: Connected to fl-iiscfg-client-0, attached to remote volume '/export/glu/web/flash/iiscfg'. [2015-06-07 09:01:04.561451] I [client-handshake.c:1203:client_setvolume_cbk] 0-fl-iiscfg-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.561419] I [client-handshake.c:1193:client_setvolume_cbk] 0-vmimage-client-1: Connected to vmimage-client-1, attached to remote volume '/export/gluster01/brick'. [2015-06-07 09:01:04.561501] I [client-handshake.c:1203:client_setvolume_cbk] 0-vmimage-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.561557] I [MSGID: 108005] [afr-common.c:3880:afr_notify] 0-fl-iiscfg-replicate-0: Subvolume 'fl-iiscfg-client-0' came back up; going online. [2015-06-07 09:01:04.561662] I [client-handshake.c:1193:client_setvolume_cbk] 0-conf-client-1: Connected to conf-client-1, attached to remote volume '/export/glu/conf'. [2015-06-07 09:01:04.561705] I [client-handshake.c:1203:client_setvolume_cbk] 0-conf-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.561646] I [MSGID: 108005] [afr-common.c:3880:afr_notify] 0-vmimage-replicate-0: Subvolume 'vmimage-client-1' came back up; going online. [2015-06-07 09:01:04.561806] I [MSGID: 108005] [afr-common.c:3880:afr_notify] 0-conf-replicate-0: Subvolume 'conf-client-1' came back up; going online. [2015-06-07 09:01:04.561876] I [client-handshake.c:1193:client_setvolume_cbk] 0-wwwdata-client-1: Connected to wwwdata-client-1, attached to remote volume '/export/glu/web/wwwdata'. [2015-06-07 09:01:04.561891] I [client-handshake.c:1193:client_setvolume_cbk] 0-fl-webroot-client-0: Connected to fl-webroot-client-0, attached to remote volume '/export/glu/web/flash/webroot'. [2015-06-07 09:01:04.561934] I [client-handshake.c:1203:client_setvolume_cbk] 0-wwwdata-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.561953] I [client-handshake.c:1203:client_setvolume_cbk] 0-fl-webroot-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.562066] I [MSGID: 108005] [afr-common.c:3880:afr_notify] 0-wwwdata-replicate-0: Subvolume 'wwwdata-client-1' came back up; going online. [2015-06-07 09:01:04.562164] I [MSGID: 108005] [afr-common.c:3880:afr_notify] 0-fl-webroot-replicate-0: Subvolume 'fl-webroot-client-0' came back up; going online. [2015-06-07 09:01:04.562236] I [client-handshake.c:1193:client_setvolume_cbk] 0-ovirt-export-client-1: Connected to ovirt-export-client-1, attached to remote volume '/export/glu/vm/ovirt-export'. [2015-06-07 09:01:04.562251] I [client-handshake.c:1193:client_setvolume_cbk] 0-IT-client-0: Connected to IT-client-0, attached to remote volume '/export/glu/common/IT'. [2015-06-07 09:01:04.562290] I [client-handshake.c:1203:client_setvolume_cbk] 0-ovirt-export-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.562306] I [client-handshake.c:1203:client_setvolume_cbk] 0-IT-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.562408] I [MSGID: 108005] [afr-common.c:3880:afr_notify] 0-IT-replicate-0: Subvolume 'IT-client-0' came back up; going online. [2015-06-07 09:01:04.562500] I [MSGID: 108005] [afr-common.c:3880:afr_notify] 0-ovirt-export-replicate-0: Subvolume 'ovirt-export-client-1' came back up; going online. [2015-06-07 09:01:04.562508] I [client-handshake.c:187:client_set_lk_version_cbk] 0-rb-webroot-client-0: Server lk version = 1 [2015-06-07 09:01:04.562627] I [client-handshake.c:187:client_set_lk_version_cbk] 0-lpx-assets-client-1: Server lk version = 1 [2015-06-07 09:01:04.562627] I [client-handshake.c:187:client_set_lk_version_cbk] 0-rb-appcfg-client-1: Server lk version = 1 [2015-06-07 09:01:04.562694] I [client-handshake.c:187:client_set_lk_version_cbk] 0-rb-iiscfg-client-0: Server lk version = 1 [2015-06-07 09:01:04.562711] I [client-handshake.c:187:client_set_lk_version_cbk] 0-mssqlp-client-1: Server lk version = 1 [2015-06-07 09:01:04.562746] I [client-handshake.c:187:client_set_lk_version_cbk] 0-fl-iiscfg-client-0: Server lk version = 1 [2015-06-07 09:01:04.562764] I [client-handshake.c:187:client_set_lk_version_cbk] 0-vmimage-client-1: Server lk version = 1 [2015-06-07 09:01:04.562796] I [client-handshake.c:187:client_set_lk_version_cbk] 0-conf-client-1: Server lk version = 1 [2015-06-07 09:01:04.562845] I [client-handshake.c:187:client_set_lk_version_cbk] 0-fl-webroot-client-0: Server lk version = 1 [2015-06-07 09:01:04.562850] I [client-handshake.c:187:client_set_lk_version_cbk] 0-wwwdata-client-1: Server lk version = 1 [2015-06-07 09:01:04.562904] I [client-handshake.c:187:client_set_lk_version_cbk] 0-IT-client-0: Server lk version = 1 [2015-06-07 09:01:04.562920] I [client-handshake.c:187:client_set_lk_version_cbk] 0-ovirt-export-client-1: Server lk version = 1 [2015-06-07 09:01:04.785331] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-IT-client-1: changing port to 49171 (from 0) [2015-06-07 09:01:04.785796] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-conf-client-0: changing port to 49159 (from 0) [2015-06-07 09:01:04.786014] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-fl-iiscfg-client-1: changing port to 49164 (from 0) [2015-06-07 09:01:04.786817] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-fl-webroot-client-1: changing port to 49167 (from 0) [2015-06-07 09:01:04.787040] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-lpx-assets-client-0: changing port to 49173 (from 0) [2015-06-07 09:01:04.787070] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-mssqlp-client-0: changing port to 49163 (from 0) [2015-06-07 09:01:04.787193] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-ovirt-export-client-0: changing port to 49170 (from 0) [2015-06-07 09:01:04.787530] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-rb-appcfg-client-0: changing port to 49169 (from 0) [2015-06-07 09:01:04.787852] I [client-handshake.c:1405:select_server_supported_programs] 0-IT-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.789044] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-rb-iiscfg-client-1: changing port to 49165 (from 0) [2015-06-07 09:01:04.789211] I [client-handshake.c:1405:select_server_supported_programs] 0-conf-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.789420] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-rb-webroot-client-1: changing port to 49166 (from 0) [2015-06-07 09:01:04.789535] I [client-handshake.c:1405:select_server_supported_programs] 0-fl-iiscfg-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.789715] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-vmimage-client-0: changing port to 49153 (from 0) [2015-06-07 09:01:04.789890] I [rpc-clnt.c:1807:rpc_clnt_reconfig] 0-wwwdata-client-0: changing port to 49172 (from 0) [2015-06-07 09:01:04.791799] I [client-handshake.c:1405:select_server_supported_programs] 0-fl-webroot-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.791981] I [client-handshake.c:1405:select_server_supported_programs] 0-lpx-assets-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.791984] I [client-handshake.c:1405:select_server_supported_programs] 0-mssqlp-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.792182] I [client-handshake.c:1405:select_server_supported_programs] 0-ovirt-export-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.792376] I [client-handshake.c:1405:select_server_supported_programs] 0-rb-appcfg-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.792435] I [client-handshake.c:1405:select_server_supported_programs] 0-rb-iiscfg-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.792522] I [client-handshake.c:1405:select_server_supported_programs] 0-vmimage-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.792646] I [client-handshake.c:1405:select_server_supported_programs] 0-rb-webroot-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.804844] I [client-handshake.c:1193:client_setvolume_cbk] 0-conf-client-0: Connected to conf-client-0, attached to remote volume '/export/glu/conf'. [2015-06-07 09:01:04.804919] I [client-handshake.c:1203:client_setvolume_cbk] 0-conf-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.807180] I [client-handshake.c:187:client_set_lk_version_cbk] 0-conf-client-0: Server lk version = 1 [2015-06-07 09:01:04.829137] I [client-handshake.c:1405:select_server_supported_programs] 0-wwwdata-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-06-07 09:01:04.829640] I [client-handshake.c:1193:client_setvolume_cbk] 0-wwwdata-client-0: Connected to wwwdata-client-0, attached to remote volume '/export/glu/web/wwwdata'. [2015-06-07 09:01:04.829687] I [client-handshake.c:1203:client_setvolume_cbk] 0-wwwdata-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.829724] I [client-handshake.c:1193:client_setvolume_cbk] 0-IT-client-1: Connected to IT-client-1, attached to remote volume '/export/glu/common/IT'. [2015-06-07 09:01:04.829774] I [client-handshake.c:1203:client_setvolume_cbk] 0-IT-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.830395] I [client-handshake.c:187:client_set_lk_version_cbk] 0-wwwdata-client-0: Server lk version = 1 [2015-06-07 09:01:04.830891] I [client-handshake.c:1193:client_setvolume_cbk] 0-lpx-assets-client-0: Connected to lpx-assets-client-0, attached to remote volume '/export/glu/web/lpx'. [2015-06-07 09:01:04.830925] I [client-handshake.c:1203:client_setvolume_cbk] 0-lpx-assets-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.836436] I [client-handshake.c:1193:client_setvolume_cbk] 0-fl-iiscfg-client-1: Connected to fl-iiscfg-client-1, attached to remote volume '/export/glu/web/flash/iiscfg'. [2015-06-07 09:01:04.836465] I [client-handshake.c:1193:client_setvolume_cbk] 0-rb-appcfg-client-0: Connected to rb-appcfg-client-0, attached to remote volume '/export/glu/web/rebuild/appcfg'. [2015-06-07 09:01:04.836479] I [client-handshake.c:1203:client_setvolume_cbk] 0-fl-iiscfg-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.836530] I [client-handshake.c:1203:client_setvolume_cbk] 0-rb-appcfg-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.836883] I [client-handshake.c:1193:client_setvolume_cbk] 0-vmimage-client-0: Connected to vmimage-client-0, attached to remote volume '/export/gluster01/brick'. [2015-06-07 09:01:04.836917] I [client-handshake.c:1203:client_setvolume_cbk] 0-vmimage-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.837171] I [client-handshake.c:1193:client_setvolume_cbk] 0-fl-webroot-client-1: Connected to fl-webroot-client-1, attached to remote volume '/export/glu/web/flash/webroot'. [2015-06-07 09:01:04.837211] I [client-handshake.c:1203:client_setvolume_cbk] 0-fl-webroot-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.837505] I [client-handshake.c:187:client_set_lk_version_cbk] 0-IT-client-1: Server lk version = 1 [2015-06-07 09:01:04.837603] I [client-handshake.c:1193:client_setvolume_cbk] 0-ovirt-export-client-0: Connected to ovirt-export-client-0, attached to remote volume '/export/glu/vm/ovirt-export'. [2015-06-07 09:01:04.837653] I [client-handshake.c:1203:client_setvolume_cbk] 0-ovirt-export-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.837728] I [client-handshake.c:187:client_set_lk_version_cbk] 0-lpx-assets-client-0: Server lk version = 1 [2015-06-07 09:01:04.837819] I [client-handshake.c:1193:client_setvolume_cbk] 0-mssqlp-client-0: Connected to mssqlp-client-0, attached to remote volume '/export/glu/db/mssql/p'. [2015-06-07 09:01:04.837837] I [client-handshake.c:1203:client_setvolume_cbk] 0-mssqlp-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.838083] I [client-handshake.c:187:client_set_lk_version_cbk] 0-rb-appcfg-client-0: Server lk version = 1 [2015-06-07 09:01:04.838196] I [client-handshake.c:187:client_set_lk_version_cbk] 0-fl-iiscfg-client-1: Server lk version = 1 [2015-06-07 09:01:04.838246] I [client-handshake.c:187:client_set_lk_version_cbk] 0-vmimage-client-0: Server lk version = 1 [2015-06-07 09:01:04.838295] I [client-handshake.c:1193:client_setvolume_cbk] 0-rb-webroot-client-1: Connected to rb-webroot-client-1, attached to remote volume '/export/glu/web/rebuild/webroot'. [2015-06-07 09:01:04.838315] I [client-handshake.c:1203:client_setvolume_cbk] 0-rb-webroot-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.858781] I [client-handshake.c:1193:client_setvolume_cbk] 0-rb-iiscfg-client-1: Connected to rb-iiscfg-client-1, attached to remote volume '/export/glu/web/rebuild/iiscfg'. [2015-06-07 09:01:04.858833] I [client-handshake.c:1203:client_setvolume_cbk] 0-rb-iiscfg-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-06-07 09:01:04.858944] I [client-handshake.c:187:client_set_lk_version_cbk] 0-fl-webroot-client-1: Server lk version = 1 [2015-06-07 09:01:04.859118] I [client-handshake.c:187:client_set_lk_version_cbk] 0-ovirt-export-client-0: Server lk version = 1 [2015-06-07 09:01:04.859199] I [client-handshake.c:187:client_set_lk_version_cbk] 0-mssqlp-client-0: Server lk version = 1 [2015-06-07 09:01:04.859220] I [client-handshake.c:187:client_set_lk_version_cbk] 0-rb-webroot-client-1: Server lk version = 1 [2015-06-07 09:01:04.859269] I [client-handshake.c:187:client_set_lk_version_cbk] 0-rb-iiscfg-client-1: Server lk version = 1 [2015-06-07 09:23:17.600126] I [afr-self-heal-common.c:651:afr_log_selfheal] 0-vmimage-replicate-0: Completed data selfheal on 950a47c7-5d0e-457f-a6da-61d2c2ba0e1c. source=1 sinks=0 [2015-06-07 09:58:03.916141] I [afr-self-heal-common.c:651:afr_log_selfheal] 0-vmimage-replicate-0: Completed data selfheal on f90e8747-ca3e-42fa-940f-3d82353c596b. source=1 sinks=0 [2015-06-07 10:43:33.505472] I [afr-self-heal-common.c:651:afr_log_selfheal] 0-vmimage-replicate-0: Completed data selfheal on 942a9620-90ce-48d6-9ea5-5a0972dc2e7e. source=1 sinks=0 [2015-06-08 01:26:02.310011] I [MSGID: 100011] [glusterfsd.c:1284:reincarnate] 0-glusterfsd: Fetching the volume file from server... [2015-06-08 01:26:02.406396] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
