Hi,
I am working on two board setup connecting to each other. Gluster version 3.7.6 is running and added two bricks in replica 2 mode but when I manually removed (detach) the one board from the setup I am getting the following error.
volume remove-brick c_glusterfs replica 1 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs
volume remove-brick c_glusterfs replica 1 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs
Please find the logs file as an attachment.
Regards,
Abhishek
[2016-02-18 10:03:34.497996] : volume set c_glusterfs nfs.disable on : SUCCESS [2016-02-18 10:03:34.915036] : volume start c_glusterfs force : SUCCESS [2016-02-18 10:03:40.250326] : volume status : SUCCESS [2016-02-18 10:03:40.273275] : volume status : SUCCESS [2016-02-18 10:03:40.601472] : volume remove-brick c_glusterfs replica 1 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS [2016-02-18 10:03:40.885973] : peer detach 10.32.1.144 : SUCCESS [2016-02-18 10:03:42.065038] : peer probe 10.32.1.144 : SUCCESS [2016-02-18 10:03:44.563546] : volume add-brick c_glusterfs replica 2 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS [2016-02-18 10:30:53.297415] : volume status : SUCCESS [2016-02-18 10:30:53.313096] : volume status : SUCCESS [2016-02-18 10:37:02.748714] : volume status : SUCCESS [2016-02-18 10:37:02.762091] : volume status : SUCCESS [2016-02-18 10:37:02.816089] : volume remove-brick c_glusterfs replica 1 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs
[2016-02-18 10:03:34.226690] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:34.239385] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:34.239523] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:34.240668] I [cli-rpc-ops.c:769:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2016-02-18 10:03:34.240937] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:34.274777] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:34.287456] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:34.287911] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:34.288414] I [cli-rpc-ops.c:769:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2016-02-18 10:03:34.288703] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:34.322680] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:34.336380] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:34.336531] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:34.498286] I [cli-rpc-ops.c:2087:gf_cli_set_volume_cbk] 0-cli: Received resp to set [2016-02-18 10:03:34.498515] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:34.532120] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:34.545314] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:34.545720] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:34.548157] I [cli-rpc-ops.c:769:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2016-02-18 10:03:34.548449] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:34.582190] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:34.595604] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:34.597094] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:34.598460] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:34.632132] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:34.645733] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:34.645868] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:34.915417] I [cli-rpc-ops.c:1348:gf_cli_start_volume_cbk] 0-cli: Received resp to start volume [2016-02-18 10:03:34.916123] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:39.711197] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:39.757052] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:39.757264] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:39.765954] I [cli-rpc-ops.c:769:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2016-02-18 10:03:39.766385] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:39.841640] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:39.876125] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:39.876278] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:39.887210] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:39.977629] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:40.005103] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:40.005284] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:40.019477] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:40.145305] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:40.230393] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:40.230557] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:40.282433] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:40.357625] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:40.387122] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:40.387300] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:40.602300] I [cli-rpc-ops.c:2695:gf_cli_remove_brick_cbk] 0-cli: Received resp to remove brick [2016-02-18 10:03:40.603099] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:40.659767] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:40.677598] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:40.678035] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:40.679200] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:40.835521] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:40.876232] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:40.877589] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:40.887335] I [cli-rpc-ops.c:189:gf_cli_deprobe_cbk] 0-cli: Received resp to deprobe [2016-02-18 10:03:40.887638] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:41.072538] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:41.127388] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:41.127700] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:41.135709] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:41.329545] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:41.351360] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:41.351604] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:41.357697] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:41.562958] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:41.579238] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:41.579386] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:42.065252] I [cli-rpc-ops.c:137:gf_cli_probe_cbk] 0-cli: Received resp to probe [2016-02-18 10:03:42.065411] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:43.101056] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:43.115036] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:43.115460] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:43.117798] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:43.155522] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:43.169147] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:43.169538] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:43.171518] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:43.208807] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:43.221111] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:43.221245] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:44.563823] I [cli-rpc-ops.c:2465:gf_cli_add_brick_cbk] 0-cli: Received resp to add brick [2016-02-18 10:03:44.563997] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:44.609222] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:44.621981] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:44.622128] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:44.624670] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:44.635924] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:44.649163] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:44.649587] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:44.654764] I [cli-rpc-ops.c:769:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2016-02-18 10:03:44.655135] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:03:44.666909] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:03:44.679001] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:44.679773] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:03:46.837333] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:04:43.681474] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:04:43.693682] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:04:43.695098] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:04:43.696078] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:04:43.729149] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:04:43.741241] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:04:43.741584] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:04:43.743526] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:06:07.197296] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:06:07.211597] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:06:07.211942] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:06:07.213938] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:06:07.246774] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:06:07.259003] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:06:07.260864] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:06:07.261264] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:08:48.629433] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:08:48.641674] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:08:48.642013] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:08:48.644254] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:08:48.677929] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:08:48.690143] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:08:48.690266] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:08:48.692584] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:12:12.728687] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:12:12.740853] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:12:12.740974] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:12:12.743189] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:12:12.778349] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:12:12.790791] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:12:12.791152] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:12:12.793884] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:14:31.534788] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:14:31.547285] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:14:31.547688] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:14:31.549696] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:14:31.583643] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:14:31.595850] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:14:31.596203] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:14:31.598593] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:14:40.834332] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:14:40.847769] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:14:40.848128] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:14:40.850071] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:14:40.884416] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:14:40.896903] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:14:40.897256] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:14:40.899460] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:51.940311] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:51.952795] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:51.953309] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:51.955890] I [cli-rpc-ops.c:769:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2016-02-18 10:30:51.956192] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:51.991159] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:52.003596] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:52.003726] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:52.005978] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:52.040205] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:52.052748] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:52.053098] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:52.055048] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:52.089696] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:52.103325] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:52.103679] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:52.105638] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:53.141611] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:53.154089] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:53.155956] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:53.156360] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:53.193767] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:53.206179] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:53.206523] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:53.208503] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:53.243778] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:53.256117] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:53.256520] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:53.313655] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:53.357467] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:53.369986] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:53.370394] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:53.372778] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:53.383779] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:53.395888] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:53.396304] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:53.398356] I [cli-rpc-ops.c:769:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2016-02-18 10:30:53.398657] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:53.410919] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:53.423044] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:53.423755] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:55.555063] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:55.780770] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:55.792957] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:55.793086] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:55.795229] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:55.806286] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:55.818510] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:55.818629] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:55.820766] I [cli-rpc-ops.c:769:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2016-02-18 10:30:55.821035] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:56.636780] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:56.648878] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:56.649224] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:56.651156] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:56.661966] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:56.674129] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:56.674496] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:56.676452] I [cli-rpc-ops.c:769:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2016-02-18 10:30:56.676695] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:59.183466] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:59.196116] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:59.196455] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:59.198549] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:30:59.210247] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:30:59.222463] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:30:59.224421] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:30:59.224779] I [cli-rpc-ops.c:769:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2016-02-18 10:30:59.225107] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:31:23.293760] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:31:23.306005] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:31:23.306140] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:31:23.308821] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:31:23.321237] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:31:23.333406] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:31:23.333779] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:31:23.335643] I [cli-rpc-ops.c:769:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2016-02-18 10:31:23.335887] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:31:24.951562] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:31:24.964072] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:31:24.964414] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:31:24.966451] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:31:24.978894] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:31:24.991778] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:31:24.991898] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:31:24.994882] I [cli-rpc-ops.c:769:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2016-02-18 10:31:24.995187] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:37:02.571660] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:37:02.583919] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:37:02.584055] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:37:02.586922] I [cli-rpc-ops.c:769:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2016-02-18 10:37:02.587208] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:37:02.622713] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:37:02.635133] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:37:02.635473] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:37:02.638271] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:37:02.673136] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:37:02.685568] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:37:02.685677] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:37:02.688098] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:37:02.725941] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:37:02.741987] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:37:02.743099] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:37:02.763358] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:37:02.800314] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:37:02.813788] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:37:02.814312] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:37:02.816278] I [cli-rpc-ops.c:2695:gf_cli_remove_brick_cbk] 0-cli: Received resp to remove brick [2016-02-18 10:37:02.816430] I [input.c:36:cli_batch] 0-: Exiting with: -1 [2016-02-18 10:37:58.014710] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:37:58.027117] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:37:58.027243] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:37:58.029380] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:37:58.064623] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:37:58.077190] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:37:58.077543] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:37:58.079441] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:38:01.331992] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:38:01.344327] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:38:01.344449] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:38:01.346768] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:38:01.381757] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:38:01.394001] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:38:01.395092] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:38:01.396847] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:38:40.492265] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:38:40.504737] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:38:40.505087] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:38:40.507616] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:38:40.542858] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:38:40.557340] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:38:40.557683] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:38:40.559785] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:39:31.853348] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:39:31.865700] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:39:31.866056] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:39:31.868012] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:39:31.902554] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:39:31.914727] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:39:31.914855] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:39:31.917137] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:42:09.633191] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:42:09.645299] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:42:09.645632] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:42:09.648194] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:42:09.683475] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:42:09.695773] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:42:09.696136] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:42:09.699262] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:42:48.031780] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:42:48.044004] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:42:48.045555] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:42:48.046900] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:42:48.082215] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:42:48.094454] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:42:48.094803] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:42:48.096710] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:43:38.041168] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:43:38.053745] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:43:38.054102] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:43:38.056095] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2016-02-18 10:43:38.090103] I [cli.c:721:main] 0-cli: Started running gluster with version 3.7.6 [2016-02-18 10:43:38.102805] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:43:38.103157] I [socket.c:2355:socket_event_handler] 0-transport: disconnecting now [2016-02-18 10:43:38.105006] I [input.c:36:cli_batch] 0-: Exiting with: 0
[2016-02-18 10:03:32.812098] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) [2016-02-18 10:03:32.820603] I [MSGID: 106478] [glusterd.c:1350:init] 0-management: Maximum allowed open file descriptors set to 65536 [2016-02-18 10:03:32.820727] I [MSGID: 106479] [glusterd.c:1399:init] 0-management: Using /system/glusterd as working directory [2016-02-18 10:03:32.839487] I [MSGID: 106513] [glusterd-store.c:2047:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2016-02-18 10:03:34.148107] I [MSGID: 106498] [glusterd-handler.c:3579:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2016-02-18 10:03:34.148345] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2016-02-18 10:03:34.150180] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 13, Invalid argument [2016-02-18 10:03:34.150218] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:03:34.152769] I [MSGID: 106544] [glusterd.c:159:glusterd_uuid_init] 0-management: retrieved UUID: a1b504ab-c13a-4d38-bf11-baf1704ac253 Final graph: +------------------------------------------------------------------------------+ [2016-02-18 10:03:34.159781] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 [2016-02-18 10:03:34.159882] E [socket.c:2278:socket_connect_finish] 0-management: connection to 10.32.1.144:24007 failed (Connection refused) 8: option ping-timeout 30 9: option transport.socket.read-fail-log off 10: option transport.socket.keepalive-interval 2 11: option transport.socket.keepalive-time 10 12: option transport-type socket 13: option working-directory /system/glusterd 14: end-volume 15: +------------------------------------------------------------------------------+ [2016-02-18 10:03:34.160086] I [MSGID: 106004] [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer <10.32.1.144> (<944907f1-6950-4f9f-bcf9-1bafaafbaea7>), in state <Peer in Cluster>, has disconnected from glusterd. [2016-02-18 10:03:34.176540] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2016-02-18 10:03:34.176590] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2016-02-18 10:03:34.240254] I [MSGID: 106488] [glusterd-handler.c:1472:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2016-02-18 10:03:34.381268] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600 [2016-02-18 10:03:34.482483] W [glusterd-nfs-svc.c:157:glusterd_nfssvc_reconfigure] 0-management: nfs_svc.name is empty using: nfs [2016-02-18 10:03:34.482529] I [glusterd-nfs-svc.c:159:glusterd_nfssvc_reconfigure] 0-management: copying nfs_svc_name to priv->nfs_svc.name: nfs [2016-02-18 10:03:34.483809] W [glusterd-shd-svc.c:203:glusterd_shdsvc_reconfigure] 0-management: shd_svc.name is empty using: glustershd [2016-02-18 10:03:34.483842] I [glusterd-shd-svc.c:205:glusterd_shdsvc_reconfigure] 0-management: copying shd_svc_name to priv->shd_svc.name: glustershd [2016-02-18 10:03:34.484327] W [dict.c:1452:dict_get_with_ref] (-->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(build_shd_graph-0xe1978) [0x3fffabd04170] -->/usr/lib64/libglusterfs.so.0(dict_get_str_boolean-0xbd2e0) [0x3fffaf36a1a8] -->/usr/lib64/libglusterfs.so.0(+0x33b84) [0x3fffaf366b84] ) 0-dict: dict OR key (graph-check) is NULL [Invalid argument] [2016-02-18 10:03:34.485936] W [dict.c:1452:dict_get_with_ref] (-->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(build_shd_graph-0xe1978) [0x3fffabd04170] -->/usr/lib64/libglusterfs.so.0(dict_get_str_boolean-0xbd2e0) [0x3fffaf36a1a8] -->/usr/lib64/libglusterfs.so.0(+0x33b84) [0x3fffaf366b84] ) 0-dict: dict OR key (graph-check) is NULL [Invalid argument] [2016-02-18 10:03:34.495399] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600 [2016-02-18 10:03:34.288121] I [MSGID: 106488] [glusterd-handler.c:1472:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2016-02-18 10:03:34.495952] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2016-02-18 10:03:34.496448] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600 [2016-02-18 10:03:34.496905] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2016-02-18 10:03:34.497364] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600 [2016-02-18 10:03:34.497818] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2016-02-18 10:03:34.547705] I [MSGID: 106488] [glusterd-handler.c:1472:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2016-02-18 10:03:34.597954] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:03:34.731838] W [common-utils.c:1685:gf_string2boolean] (-->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(glusterd_op_commit_perform-0x135074) [0x3fffabcad7bc] -->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(glusterd_op_start_volume-0xb1c8c) [0x3fffabd3502c] -->/usr/lib64/libglusterfs.so.0(gf_string2boolean-0x96cd8) [0x3fffaf393018] ) 0-management: argument invalid [Invalid argument] [2016-02-18 10:03:34.808705] I [MSGID: 106143] [glusterd-pmap.c:229:pmap_registry_bind] 0-pmap: adding brick /opt/lvmdir/c2/brick on port 49173 [2016-02-18 10:03:34.811395] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2016-02-18 10:03:34.901135] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600 [2016-02-18 10:03:34.901771] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2016-02-18 10:03:34.902596] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600 [2016-02-18 10:03:34.904622] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped [2016-02-18 10:03:34.913935] W [socket.c:3009:socket_connect] 0-glustershd: Ignore failed connection attempt on , (No such file or directory) [2016-02-18 10:03:34.914417] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2016-02-18 10:03:34.914764] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2016-02-18 10:03:34.918164] W [socket.c:588:__socket_rwv] 0-glustershd: readv on /var/run/gluster/6ad009e3d67de47b2f44f2d196b44675.socket failed (Invalid argument) [2016-02-18 10:03:34.918209] I [MSGID: 106006] [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd. [2016-02-18 10:03:37.818794] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:03:37.818840] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:03:38.194788] I [MSGID: 106163] [glusterd-handshake.c:1193:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2016-02-18 10:03:38.228034] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 944907f1-6950-4f9f-bcf9-1bafaafbaea7 [2016-02-18 10:03:38.229299] E [MSGID: 101076] [common-utils.c:2954:gf_get_hostname_from_ip] 0-common-utils: Could not lookup hostname of 10.32.1.144 : Temporary failure in name resolution [2016-02-18 10:03:38.242929] I [MSGID: 106493] [glusterd-handler.c:3780:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.32.1.144 (0), ret: 0 [2016-02-18 10:03:38.274040] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2016-02-18 10:03:39.290471] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped [2016-02-18 10:03:39.290739] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2016-02-18 10:03:39.290909] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2016-02-18 10:03:39.291641] I [MSGID: 106493] [glusterd-rpc-ops.c:480:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 944907f1-6950-4f9f-bcf9-1bafaafbaea7, host: 10.32.1.144, port: 0 [2016-02-18 10:03:39.308447] I [MSGID: 106492] [glusterd-handler.c:2715:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 944907f1-6950-4f9f-bcf9-1bafaafbaea7 [2016-02-18 10:03:39.308672] I [MSGID: 106502] [glusterd-handler.c:2760:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2016-02-18 10:03:39.309001] I [MSGID: 106006] [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd. [2016-02-18 10:03:39.480672] I [MSGID: 106493] [glusterd-rpc-ops.c:694:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 944907f1-6950-4f9f-bcf9-1bafaafbaea7 [2016-02-18 10:03:39.761127] I [MSGID: 106488] [glusterd-handler.c:1472:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2016-02-18 10:03:39.885770] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:03:40.018754] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:03:40.256130] I [MSGID: 106499] [glusterd-handler.c:4267:__glusterd_handle_status_volume] 0-management: Received status volume req for volume c_glusterfs [2016-02-18 10:03:40.393366] I [MSGID: 106484] [glusterd-brick-ops.c:836:__glusterd_handle_remove_brick] 0-management: Received rem brick req [2016-02-18 10:03:40.393556] I [MSGID: 106062] [glusterd-brick-ops.c:914:__glusterd_handle_remove_brick] 0-management: request to change replica-count to 1 [2016-02-18 10:03:40.408106] I [MSGID: 106062] [glusterd-op-sm.c:5745:glusterd_bricks_select_remove_brick] 0-management: force flag is not set [2016-02-18 10:03:40.415947] W [dict.c:592:dict_unref] (-->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(glusterd_op_commit_perform-0x1350d4) [0x3fffabcad75c] -->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(glusterd_op_remove_brick-0xa98b0) [0x3fffabd3d6c0] -->/usr/lib64/libglusterfs.so.0(dict_unref-0xc08c0) [0x3fffaf366520] ) 0-dict: dict is NULL [Invalid argument] [2016-02-18 10:03:40.416478] I [MSGID: 106062] [glusterd-brick-ops.c:2556:glusterd_op_remove_brick] 0-management: changing replica count 2 to 1 on volume c_glusterfs [2016-02-18 10:03:40.510462] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2016-02-18 10:03:40.511993] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2016-02-18 10:03:40.512284] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2016-02-18 10:03:40.678593] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:03:40.884156] I [MSGID: 106487] [glusterd-handler.c:1290:__glusterd_handle_cli_deprobe] 0-glusterd: Received CLI deprobe req [2016-02-18 10:03:40.885869] I [MSGID: 106493] [glusterd-rpc-ops.c:598:__glusterd_friend_remove_cbk] 0-glusterd: Received ACC from uuid: 944907f1-6950-4f9f-bcf9-1bafaafbaea7, host: 10.32.0.48, port: 0 [2016-02-18 10:03:41.132881] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:03:41.355319] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:03:41.584164] I [MSGID: 106487] [glusterd-handler.c:1178:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 10.32.1.144 24007 [2016-02-18 10:03:41.586896] I [MSGID: 106129] [glusterd-handler.c:3600:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: 10.32.1.144 (24007) [2016-02-18 10:03:41.616187] I [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2016-02-18 10:03:41.617179] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 18, Invalid argument [2016-02-18 10:03:41.617219] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:03:41.617494] I [MSGID: 106498] [glusterd-handler.c:3528:glusterd_friend_add] 0-management: connect returned 0 [2016-02-18 10:03:41.945661] I [MSGID: 106511] [glusterd-rpc-ops.c:256:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 944907f1-6950-4f9f-bcf9-1bafaafbaea7, host: 10.32.1.144 [2016-02-18 10:03:41.945747] I [MSGID: 106511] [glusterd-rpc-ops.c:416:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2016-02-18 10:03:42.064937] I [MSGID: 106493] [glusterd-rpc-ops.c:480:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 944907f1-6950-4f9f-bcf9-1bafaafbaea7, host: 10.32.1.144, port: 0 [2016-02-18 10:03:42.103756] I [MSGID: 106163] [glusterd-handshake.c:1193:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2016-02-18 10:03:42.150048] I [MSGID: 106490] [glusterd-handler.c:2884:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 944907f1-6950-4f9f-bcf9-1bafaafbaea7 [2016-02-18 10:03:42.150661] E [MSGID: 101076] [common-utils.c:2954:gf_get_hostname_from_ip] 0-common-utils: Could not lookup hostname of 10.32.1.144 : Temporary failure in name resolution [2016-02-18 10:03:42.150804] I [MSGID: 106493] [glusterd-handler.c:2947:__glusterd_handle_probe_query] 0-glusterd: Responded to 10.32.1.144, op_ret: 0, op_errno: 0, ret: 0 [2016-02-18 10:03:42.167001] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 944907f1-6950-4f9f-bcf9-1bafaafbaea7 [2016-02-18 10:03:42.167462] E [MSGID: 101076] [common-utils.c:2954:gf_get_hostname_from_ip] 0-common-utils: Could not lookup hostname of 10.32.1.144 : Temporary failure in name resolution [2016-02-18 10:03:42.182203] I [MSGID: 106493] [glusterd-handler.c:3780:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.32.1.144 (0), ret: 0 [2016-02-18 10:03:42.215548] I [MSGID: 106493] [glusterd-rpc-ops.c:694:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 944907f1-6950-4f9f-bcf9-1bafaafbaea7 [2016-02-18 10:03:42.215735] I [MSGID: 106492] [glusterd-handler.c:2715:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 944907f1-6950-4f9f-bcf9-1bafaafbaea7 [2016-02-18 10:03:42.215802] I [MSGID: 106502] [glusterd-handler.c:2760:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2016-02-18 10:03:43.117159] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:03:43.171075] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:03:43.223315] I [MSGID: 106482] [glusterd-brick-ops.c:442:__glusterd_handle_add_brick] 0-management: Received add brick req [2016-02-18 10:03:43.223424] I [MSGID: 106062] [glusterd-brick-ops.c:494:__glusterd_handle_add_brick] 0-management: replica-count is 2 [2016-02-18 10:03:43.223459] I [MSGID: 106447] [glusterd-brick-ops.c:240:gd_addbr_validate_replica_count] 0-management: Changing the type of volume c_glusterfs from 'distribute' to 'replica' [2016-02-18 10:03:43.233486] I [MSGID: 106062] [glusterd-brick-ops.c:1268:glusterd_op_perform_add_bricks] 0-management: replica-count is set 2 [2016-02-18 10:03:43.233550] I [MSGID: 106062] [glusterd-brick-ops.c:1273:glusterd_op_perform_add_bricks] 0-management: type is set 2, need to change it [2016-02-18 10:03:43.237661] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped [2016-02-18 10:03:43.248264] W [socket.c:588:__socket_rwv] 0-glustershd: readv on /var/run/gluster/6ad009e3d67de47b2f44f2d196b44675.socket failed (No data available) [2016-02-18 10:03:44.259057] W [socket.c:3009:socket_connect] 0-glustershd: Ignore failed connection attempt on /var/run/gluster/6ad009e3d67de47b2f44f2d196b44675.socket, (No such file or directory) [2016-02-18 10:03:44.259717] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped [2016-02-18 10:03:44.259954] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped [2016-02-18 10:03:44.260835] W [socket.c:588:__socket_rwv] 0-glustershd: readv on /var/run/gluster/6ad009e3d67de47b2f44f2d196b44675.socket failed (Invalid argument) [2016-02-18 10:03:44.260969] I [MSGID: 106006] [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify] 0-management: glustershd has disconnected from glusterd. [2016-02-18 10:03:44.624221] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:03:44.654311] I [MSGID: 106488] [glusterd-handler.c:1472:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2016-02-18 10:04:08.828555] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-management: size=588 max=3 total=17 [2016-02-18 10:04:08.828627] I [MSGID: 101053] [mem-pool.c:616:mem_pool_destroy] 0-management: size=124 max=3 total=17 [2016-02-18 10:04:43.695594] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:04:43.743106] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:06:07.213449] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:06:07.260868] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:08:48.643520] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:08:48.692107] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:12:12.742695] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:12:12.792655] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:14:31.549230] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req The message "I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 3 times between [2016-02-18 10:14:31.549230] and [2016-02-18 10:14:40.898956] [2016-02-18 10:28:42.033110] W [socket.c:588:__socket_rwv] 0-management: readv on 10.32.1.144:24007 failed (Connection timed out) [2016-02-18 10:28:42.033213] I [MSGID: 106004] [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer <10.32.1.144> (<944907f1-6950-4f9f-bcf9-1bafaafbaea7>), in state <Peer in Cluster>, has disconnected from glusterd. [2016-02-18 10:28:42.033730] W [glusterd-locks.c:681:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(glusterd_big_locked_notify-0x14b5a0) [0x3fffabc964f8] -->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify-0x141264) [0x3fffabca0f54] -->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock-0x9dd2c) [0x3fffabd4997c] ) 0-management: Lock for vol c_glusterfs not held [2016-02-18 10:28:42.033763] W [MSGID: 106118] [glusterd-handler.c:5087:__glusterd_peer_rpc_notify] 0-management: Lock not released for c_glusterfs [2016-02-18 10:28:52.051945] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:28:52.051989] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:10.083198] E [socket.c:2278:socket_connect_finish] 0-management: connection to 10.32.1.144:24007 failed (No route to host) [2016-02-18 10:29:13.055680] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:13.055741] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:16.056875] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:16.056940] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:19.058093] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:19.058151] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:22.059301] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:22.059346] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:25.060458] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:25.060505] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:28.061616] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:28.061662] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:31.062816] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:31.062863] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:34.063996] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:34.064084] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:37.065233] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:37.065280] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:40.066435] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:40.066480] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument The message "I [MSGID: 106004] [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer <10.32.1.144> (<944907f1-6950-4f9f-bcf9-1bafaafbaea7>), in state <Peer in Cluster>, has disconnected from glusterd." repeated 11 times between [2016-02-18 10:28:42.033213] and [2016-02-18 10:29:40.625464] [2016-02-18 10:29:43.067774] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:43.067852] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:43.631457] I [MSGID: 106004] [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer <10.32.1.144> (<944907f1-6950-4f9f-bcf9-1bafaafbaea7>), in state <Peer in Cluster>, has disconnected from glusterd. [2016-02-18 10:29:46.068980] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:46.069049] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:49.070153] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:49.070200] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:52.071356] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:52.071400] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:55.072493] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:55.072538] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:29:58.073663] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:29:58.073710] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:01.074871] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:01.074917] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:04.076105] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:04.076151] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:07.077285] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:07.077359] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:10.078437] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:10.078482] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:13.079580] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:13.079627] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:16.080729] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:16.080775] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:19.081849] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:19.081896] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:22.083067] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:22.083112] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:25.084195] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:25.084241] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:28.085514] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:28.085595] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:31.086684] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:31.086730] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:34.087838] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:34.087894] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:37.088989] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:37.089048] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:40.090142] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:40.090188] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:43.091316] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:43.091362] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:49.092966] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:30:49.093013] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:30:51.781554] E [MSGID: 101076] [common-utils.c:2954:gf_get_hostname_from_ip] 0-common-utils: Could not lookup hostname of 10.32.1.144 : Temporary failure in name resolution [2016-02-18 10:30:51.783347] I [MSGID: 106163] [glusterd-handshake.c:1193:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2016-02-18 10:30:51.798534] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 6adf57dc-c619-4e56-ae40-90e6aef75fe9 [2016-02-18 10:30:51.954975] I [MSGID: 106488] [glusterd-handler.c:1472:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2016-02-18 10:30:51.969391] I [MSGID: 106493] [glusterd-handler.c:3780:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.32.1.144 (0), ret: 0 [2016-02-18 10:30:52.005460] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:30:52.094156] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 16, Invalid argument [2016-02-18 10:30:52.094195] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument The message "I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 4 times between [2016-02-18 10:30:52.005460] and [2016-02-18 10:30:53.208086] [2016-02-18 10:30:53.258422] I [MSGID: 106493] [glusterd-rpc-ops.c:480:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 6adf57dc-c619-4e56-ae40-90e6aef75fe9, host: 10.32.1.144, port: 0 [2016-02-18 10:30:53.274571] I [MSGID: 106492] [glusterd-handler.c:2715:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 6adf57dc-c619-4e56-ae40-90e6aef75fe9 [2016-02-18 10:30:53.274685] I [MSGID: 106502] [glusterd-handler.c:2760:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2016-02-18 10:30:53.299947] I [MSGID: 106499] [glusterd-handler.c:4267:__glusterd_handle_status_volume] 0-management: Received status volume req for volume c_glusterfs [2016-02-18 10:30:53.372304] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:30:53.397875] I [MSGID: 106488] [glusterd-handler.c:1472:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2016-02-18 10:30:55.794820] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:30:55.820418] I [MSGID: 106488] [glusterd-handler.c:1472:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2016-02-18 10:30:56.650754] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:30:56.676082] I [MSGID: 106488] [glusterd-handler.c:1472:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2016-02-18 10:30:59.198054] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:30:59.224391] I [MSGID: 106488] [glusterd-handler.c:1472:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2016-02-18 10:31:23.308142] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:31:23.335296] I [MSGID: 106488] [glusterd-handler.c:1472:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2016-02-18 10:31:24.965975] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:31:24.993797] I [MSGID: 106488] [glusterd-handler.c:1472:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req The message "I [MSGID: 106004] [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer <10.32.1.144> (<944907f1-6950-4f9f-bcf9-1bafaafbaea7>), in state <Peer in Cluster>, has disconnected from glusterd." repeated 21 times between [2016-02-18 10:29:43.631457] and [2016-02-18 10:30:49.093596] [2016-02-18 10:30:51.799231] E [MSGID: 101076] [common-utils.c:2954:gf_get_hostname_from_ip] 0-common-utils: Could not lookup hostname of 10.32.1.144 : Temporary failure in name resolution [2016-02-18 10:33:54.037198] W [socket.c:588:__socket_rwv] 0-management: readv on 10.32.1.144:24007 failed (No data available) [2016-02-18 10:33:54.037297] I [MSGID: 106004] [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer <10.32.1.144> (<6adf57dc-c619-4e56-ae40-90e6aef75fe9>), in state <Peer in Cluster>, has disconnected from glusterd. [2016-02-18 10:33:54.037607] W [glusterd-locks.c:681:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(glusterd_big_locked_notify-0x14b5a0) [0x3fffabc964f8] -->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(__glusterd_peer_rpc_notify-0x141264) [0x3fffabca0f54] -->/usr/lib64/glusterfs/3.7.6/xlator/mgmt/glusterd.so(glusterd_mgmt_v3_unlock-0x9dd2c) [0x3fffabd4997c] ) 0-management: Lock for vol c_glusterfs not held [2016-02-18 10:33:54.037638] W [MSGID: 106118] [glusterd-handler.c:5087:__glusterd_peer_rpc_notify] 0-management: Lock not released for c_glusterfs [2016-02-18 10:34:04.112735] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:34:04.112781] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:34:35.173393] E [socket.c:2278:socket_connect_finish] 0-management: connection to 10.32.1.144:24007 failed (Connection refused) [2016-02-18 10:34:37.116698] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:34:37.116744] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:34:40.117831] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:34:40.117876] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:34:43.118927] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:34:43.118973] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:35:19.123057] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:35:19.123104] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:35:22.124194] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:35:22.124240] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:35:25.125321] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:35:25.125367] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:35:31.126686] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:35:31.126736] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:35:34.127861] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:35:34.127906] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:35:37.129003] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:35:37.129058] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:35:40.130170] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:35:40.130217] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument The message "I [MSGID: 106004] [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer <10.32.1.144> (<6adf57dc-c619-4e56-ae40-90e6aef75fe9>), in state <Peer in Cluster>, has disconnected from glusterd." repeated 11 times between [2016-02-18 10:33:54.037297] and [2016-02-18 10:35:40.130802] [2016-02-18 10:35:43.131337] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:35:43.131382] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:35:43.131945] I [MSGID: 106004] [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer <10.32.1.144> (<6adf57dc-c619-4e56-ae40-90e6aef75fe9>), in state <Peer in Cluster>, has disconnected from glusterd. [2016-02-18 10:35:46.132484] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:35:46.132533] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:35:49.133631] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:35:49.133677] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:35:52.134772] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:35:52.134817] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:35:55.135911] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:35:55.135958] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:35:58.137091] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:35:58.137137] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:01.138243] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:01.138289] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:04.139398] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:04.139440] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:07.140565] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:07.140619] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:10.141681] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:10.141722] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:13.142787] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:13.142830] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:16.144055] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:16.144101] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:19.145239] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:19.145286] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:22.146383] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:22.146427] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:25.147505] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:25.147551] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:28.148666] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:28.148711] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:31.149746] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:31.149791] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:34.150850] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:34.150894] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:37.151959] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:37.152006] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:40.153187] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:40.153232] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:43.154301] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:43.154345] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:46.155441] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:46.155487] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:49.156578] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:49.156624] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:52.157669] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:52.157715] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:55.158761] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:55.158816] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:36:58.159877] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:36:58.159921] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:37:01.161001] W [socket.c:869:__socket_keepalive] 0-socket: failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument [2016-02-18 10:37:01.161066] E [socket.c:2965:socket_connect] 0-management: Failed to set keep-alive: Invalid argument [2016-02-18 10:37:02.270436] I [MSGID: 106163] [glusterd-handshake.c:1193:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2016-02-18 10:37:02.307416] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 6adf57dc-c619-4e56-ae40-90e6aef75fe9 [2016-02-18 10:37:02.307825] E [MSGID: 101076] [common-utils.c:2954:gf_get_hostname_from_ip] 0-common-utils: Could not lookup hostname of 10.32.1.144 : Temporary failure in name resolution [2016-02-18 10:37:02.321134] I [MSGID: 106493] [glusterd-handler.c:3780:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.32.1.144 (0), ret: 0 [2016-02-18 10:37:02.349919] I [MSGID: 106493] [glusterd-rpc-ops.c:480:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 6adf57dc-c619-4e56-ae40-90e6aef75fe9, host: 10.32.1.144, port: 0 [2016-02-18 10:37:02.375696] I [MSGID: 106492] [glusterd-handler.c:2715:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 6adf57dc-c619-4e56-ae40-90e6aef75fe9 [2016-02-18 10:37:02.375788] I [MSGID: 106502] [glusterd-handler.c:2760:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend [2016-02-18 10:37:02.396871] I [MSGID: 106493] [glusterd-rpc-ops.c:694:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 6adf57dc-c619-4e56-ae40-90e6aef75fe9 [2016-02-18 10:37:02.585909] I [MSGID: 106488] [glusterd-handler.c:1472:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req [2016-02-18 10:37:02.636988] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:37:02.687444] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2016-02-18 10:37:02.750899] I [MSGID: 106499] [glusterd-handler.c:4267:__glusterd_handle_status_volume] 0-management: Received status volume req for volume c_glusterfs [2016-02-18 10:37:02.815826] I [MSGID: 106484] [glusterd-brick-ops.c:836:__glusterd_handle_remove_brick] 0-management: Received rem brick req [2016-02-18 10:37:02.815909] I [MSGID: 106062] [glusterd-brick-ops.c:914:__glusterd_handle_remove_brick] 0-management: request to change replica-count to 1 [2016-02-18 10:37:02.816009] E [MSGID: 106256] [glusterd-brick-ops.c:1047:__glusterd_handle_remove_brick] 0-management: Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs [Invalid argument] [2016-02-18 10:37:02.816061] E [MSGID: 106265] [glusterd-brick-ops.c:1088:__glusterd_handle_remove_brick] 0-management: Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs The message "I [MSGID: 106004] [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer <10.32.1.144> (<6adf57dc-c619-4e56-ae40-90e6aef75fe9>), in state <Peer in Cluster>, has disconnected from glusterd." repeated 25 times between [2016-02-18 10:35:43.131945] and [2016-02-18 10:36:58.160458] [2016-02-18 10:37:58.028978] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req The message "I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 7 times between [2016-02-18 10:37:58.028978] and [2016-02-18 10:39:31.916637] [2016-02-18 10:42:09.647160] I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req The message "I [MSGID: 106487] [glusterd-handler.c:1411:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 5 times between [2016-02-18 10:42:09.647160] and [2016-02-18 10:43:38.104660]
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://www.gluster.org/mailman/listinfo/gluster-users
