Upgraded to 3.7.1, now glusterd core dumps on 1 server

"Freer, Eva B." <freereb@xxxxxxxx> · Sat, 6 Jun 2015 04:57:35 +0000

We have a distributed, replicated configuration with 2 servers. Both servers are running RHEL 6.6 and were fully patched on 5/27/2015. At that time I upgraded to Gluster 3.7.0. 

One server uses Infiniband to supply data to 6 clients. I also updated the Mellanox OFED drivers, which removed rdma and glusterfs-rdma saying they are no longer needed, on all the clients
 last week. All seemed to be well. 

This configuration serves 2 volumes: home and data. The home volume is small and has plenty of space. The data volume consisted of 5 x 33TB bricks on each server which are replicated via
 Ethernet. We were at 95% capacity and the long awaited storage expansion was put into service this week with the addition of 2 x 50TB bricks on each server.

After adding the bricks, I did: 'gluster volume rebalance data fix-layout start'
This failed due to no longer having the rdma, so I removed the rdma transport type using: 'gluster volume set data config.transport tcp'
I was then able to successfully run the rebalance fix-layout.

After this completed, I attempted to do a full rebalance with: 'gluster volume rebalance data start'
This did not seem to be going well with the following or similar line repeatedly in the data-rebalance.log:

[2015-06-05 23:06:14.306519] E [socket.c:2913:socket_connect] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x3beb224fb0] (--> /usr/lib64/glusterfs/3.7.0/rpc-transport/socket.so(+0x7821)[0x7f4f5fbed821] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_reconnect+0xd9)[0x3beb60e729]
 (--> /usr/lib64/libglusterfs.so.0(gf_timer_proc+0x113)[0x3beb245573] (--> /lib64/libpthread.so.0[0x3be9e079d1] ))))) 0-socket: invalid argument: this->private

I stopped the rebalance with: 'gluster volume rebalance data stop'
It appeared to stop, and 'gluster volume rebalance data status' reported that no rebalance was running. Subsequent attempts to restart it failed, reporting that rebalance was already running. 

Hoping to solve the problem, I unmounted the volumes on all clients, stopped both volumes, then stopped glusterfsd and glusterd using the init.d scripts. There were still a few gluster
 processes hanging around on one server, so I killed them with a SIGTERM. Hoping that the problem would be fixed by the 3.7.1 update, I updated all the gluster packages on both servers.

After the update, glusterd starts on one server (svr2 below), but core dumps every time on the other (svr1 below). I attempted to downgrade back to 3.7.0, but yum tells me that the packages aren't downgradable. Both servers have
 also been rebooted.

The node-state.info file on svr1 which is failing has:

rebalance_status=1
status=1
rebalance_op=19
rebalance-id=fac3a496-34b4-42e8-889c-60ad0c4d6fd1

On svr2, which is working, it has:

rebalance_status=1
status=4
rebalance_op=0
rebalance-id=00000000-0000-0000-0000-000000000000

I believe that the problem is related to the failed rebalance, but could really use some help in resolving it.

TIA,
Eva

 I ran 'glusterd —debug' and got the following output below:

[root@svr1 ~]# glusterd --debug
[2015-06-06 04:23:53.382235] I [MSGID: 100030] [glusterfsd.c:2294:main] 0-glusterd: Started running glusterd version 3.7.1 (args: glusterd --debug)
[2015-06-06 04:23:53.382286] D [logging.c:1788:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5
[2015-06-06 04:23:53.382471] D [MSGID: 0] [glusterfsd.c:626:get_volfp] 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol
[2015-06-06 04:23:53.385879] I [glusterd.c:1282:init] 0-management: Maximum allowed open file descriptors set to 65536
[2015-06-06 04:23:53.385909] I [glusterd.c:1327:init] 0-management: Using /var/lib/glusterd as working directory
[2015-06-06 04:23:53.385965] D [glusterd.c:391:glusterd_rpcsvc_options_build] 0-: listen-backlog value: 128
[2015-06-06 04:23:53.386118] D [rpcsvc.c:2315:rpcsvc_init] 0-rpc-service: RPC service inited.
[2015-06-06 04:23:53.386134] D [rpcsvc.c:1872:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
[2015-06-06 04:23:53.386170] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
[2015-06-06 04:23:53.391242] D [socket.c:3867:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0
[2015-06-06 04:23:53.391266] D [socket.c:3938:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled
[2015-06-06 04:23:53.391274] D [socket.c:3941:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled
[2015-06-06 04:23:53.391282] D [socket.c:3958:socket_init] 0-socket.management: using system polling thread
[2015-06-06 04:23:53.391294] D [name.c:550:server_fill_address_family] 0-socket.management: option address-family not specified, defaulting to inet
[2015-06-06 04:23:53.391388] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/rdma.so
[2015-06-06 04:23:53.391421] E [rpc-transport.c:291:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.7.1/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2015-06-06 04:23:53.391436] W [rpc-transport.c:295:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2015-06-06 04:23:53.391445] W [rpcsvc.c:1595:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2015-06-06 04:23:53.391453] E [glusterd.c:1515:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2015-06-06 04:23:53.391462] D [rpcsvc.c:1872:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437, Ver: 2, Port: 0
[2015-06-06 04:23:53.391471] D [rpcsvc.c:1872:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli read-only, Num: 1238463, Ver: 2, Port: 0
[2015-06-06 04:23:53.391479] D [rpcsvc.c:1872:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433, Ver: 2, Port: 0
[2015-06-06 04:23:53.391487] D [rpcsvc.c:1872:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433, Ver: 3, Port: 0
[2015-06-06 04:23:53.391496] D [rpcsvc.c:1872:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver: 1, Port: 0
[2015-06-06 04:23:53.391504] D [rpcsvc.c:1872:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633, Ver: 2, Port: 0
[2015-06-06 04:23:53.391512] D [rpcsvc.c:1872:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster MGMT Handshake, Num: 1239873, Ver: 1, Port: 0
[2015-06-06 04:23:53.391548] D [rpcsvc.c:2315:rpcsvc_init] 0-rpc-service: RPC service inited.
[2015-06-06 04:23:53.391558] D [rpcsvc.c:1872:rpcsvc_program_register] 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1, Port: 0
[2015-06-06 04:23:53.391574] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
[2015-06-06 04:23:53.391649] D [socket.c:3816:socket_init] 0-socket.management: disabling nodelay
[2015-06-06 04:23:53.391660] D [socket.c:3867:socket_init] 0-socket.management: Configued transport.tcp-user-timeout=0
[2015-06-06 04:23:53.391670] D [socket.c:3938:socket_init] 0-socket.management: SSL support on the I/O path is NOT enabled
[2015-06-06 04:23:53.391677] D [socket.c:3941:socket_init] 0-socket.management: SSL support for glusterd is NOT enabled
[2015-06-06 04:23:53.391685] D [socket.c:3958:socket_init] 0-socket.management: using system polling thread
[2015-06-06 04:23:53.391764] D [rpcsvc.c:1872:rpcsvc_program_register] 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver: 2, Port: 0
[2015-06-06 04:23:53.391779] D [rpcsvc.c:1872:rpcsvc_program_register] 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec), Num: 14398633, Ver: 2, Port: 0
[2015-06-06 04:23:53.391814] D [glusterd-utils.c:5763:glusterd_sm_tr_log_init] 0-: returning 0
[2015-06-06 04:23:53.391827] D [glusterd.c:1582:init] 0-management: cannot get run-with-valgrind value
[2015-06-06 04:23:53.540315] D [glusterd.c:446:glusterd_check_gsync_present] 0-glusterd: Returning 0
[2015-06-06 04:23:53.540446] D [glusterd.c:572:glusterd_crt_georep_folders] 0-: Returning 0
[2015-06-06 04:23:57.900056] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:57.900106] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:57.900179] D [store.c:304:gf_store_retrieve_value] 0-: key operating-version found
[2015-06-06 04:23:57.900222] I [glusterd-store.c:1986:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30600
[2015-06-06 04:23:57.900306] D [store.c:304:gf_store_retrieve_value] 0-: key UUID found
[2015-06-06 04:23:57.900355] D [glusterd-store.c:2055:glusterd_retrieve_uuid] 0-: Returning 0
[2015-06-06 04:23:57.900386] I [glusterd.c:154:glusterd_uuid_init] 0-management: retrieved UUID: 0512eb62-b9d1-4f28-9e61-52bc4c50902b
[2015-06-06 04:23:57.900502] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2015-06-06 04:23:57.900520] D [rpc-clnt.c:989:rpc_clnt_connection_init] 0-glustershd: disable ping-timeout
[2015-06-06 04:23:57.900539] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
[2015-06-06 04:23:57.900696] D [socket.c:3816:socket_init] 0-glustershd: disabling nodelay
[2015-06-06 04:23:57.900715] D [socket.c:3867:socket_init] 0-glustershd: Configued transport.tcp-user-timeout=0
[2015-06-06 04:23:57.900733] D [socket.c:3938:socket_init] 0-glustershd: SSL support on the I/O path is NOT enabled
[2015-06-06 04:23:57.900746] D [socket.c:3941:socket_init] 0-glustershd: SSL support for glusterd is NOT enabled
[2015-06-06 04:23:57.900759] D [socket.c:3958:socket_init] 0-glustershd: using system polling thread
[2015-06-06 04:23:57.900817] D [MSGID: 0] [glusterd-svc-mgmt.c:116:glusterd_svc_init_common] 0-management: Returning 0
[2015-06-06 04:23:57.900843] D [glusterd.c:1204:glusterd_svc_init_all] 0-management: shd service initialized
[2015-06-06 04:23:57.900914] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2015-06-06 04:23:57.900930] D [rpc-clnt.c:989:rpc_clnt_connection_init] 0-nfs: disable ping-timeout
[2015-06-06 04:23:57.900997] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
[2015-06-06 04:23:57.901120] D [socket.c:3816:socket_init] 0-nfs: disabling nodelay
[2015-06-06 04:23:57.901137] D [socket.c:3867:socket_init] 0-nfs: Configued transport.tcp-user-timeout=0
[2015-06-06 04:23:57.901153] D [socket.c:3938:socket_init] 0-nfs: SSL support on the I/O path is NOT enabled
[2015-06-06 04:23:57.901165] D [socket.c:3941:socket_init] 0-nfs: SSL support for glusterd is NOT enabled
[2015-06-06 04:23:57.901177] D [socket.c:3958:socket_init] 0-nfs: using system polling thread
[2015-06-06 04:23:57.901220] D [glusterd.c:1212:glusterd_svc_init_all] 0-management: nfs service initialized
[2015-06-06 04:23:57.901291] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2015-06-06 04:23:57.901307] D [rpc-clnt.c:989:rpc_clnt_connection_init] 0-quotad: disable ping-timeout
[2015-06-06 04:23:57.901324] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
[2015-06-06 04:23:57.901449] D [socket.c:3816:socket_init] 0-quotad: disabling nodelay
[2015-06-06 04:23:57.901466] D [socket.c:3867:socket_init] 0-quotad: Configued transport.tcp-user-timeout=0
[2015-06-06 04:23:57.901482] D [socket.c:3938:socket_init] 0-quotad: SSL support on the I/O path is NOT enabled
[2015-06-06 04:23:57.901494] D [socket.c:3941:socket_init] 0-quotad: SSL support for glusterd is NOT enabled
[2015-06-06 04:23:57.901506] D [socket.c:3958:socket_init] 0-quotad: using system polling thread
[2015-06-06 04:23:57.901553] D [glusterd.c:1221:glusterd_svc_init_all] 0-management: quotad service initialized
[2015-06-06 04:23:57.901620] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2015-06-06 04:23:57.901636] D [rpc-clnt.c:989:rpc_clnt_connection_init] 0-bitd: disable ping-timeout
[2015-06-06 04:23:57.901656] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
[2015-06-06 04:23:57.901775] D [socket.c:3816:socket_init] 0-bitd: disabling nodelay
[2015-06-06 04:23:57.901792] D [socket.c:3867:socket_init] 0-bitd: Configued transport.tcp-user-timeout=0
[2015-06-06 04:23:57.901807] D [socket.c:3938:socket_init] 0-bitd: SSL support on the I/O path is NOT enabled
[2015-06-06 04:23:57.901819] D [socket.c:3941:socket_init] 0-bitd: SSL support for glusterd is NOT enabled
[2015-06-06 04:23:57.901832] D [socket.c:3958:socket_init] 0-bitd: using system polling thread
[2015-06-06 04:23:57.901873] D [glusterd.c:1230:glusterd_svc_init_all] 0-management: BitD service initialized
[2015-06-06 04:23:57.901955] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2015-06-06 04:23:57.901974] D [rpc-clnt.c:989:rpc_clnt_connection_init] 0-scrub: disable ping-timeout
[2015-06-06 04:23:57.901995] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
[2015-06-06 04:23:57.902115] D [socket.c:3816:socket_init] 0-scrub: disabling nodelay
[2015-06-06 04:23:57.902132] D [socket.c:3867:socket_init] 0-scrub: Configued transport.tcp-user-timeout=0
[2015-06-06 04:23:57.902147] D [socket.c:3938:socket_init] 0-scrub: SSL support on the I/O path is NOT enabled
[2015-06-06 04:23:57.902166] D [socket.c:3941:socket_init] 0-scrub: SSL support for glusterd is NOT enabled
[2015-06-06 04:23:57.902179] D [socket.c:3958:socket_init] 0-scrub: using system polling thread
[2015-06-06 04:23:57.902221] D [glusterd.c:1238:glusterd_svc_init_all] 0-management: scrub service initialized
[2015-06-06 04:23:57.902310] D [glusterd-utils.c:519:glusterd_volinfo_new] 0-management: Returning 0
[2015-06-06 04:23:57.902388] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2015-06-06 04:23:57.902405] D [rpc-clnt.c:989:rpc_clnt_connection_init] 0-snapd: disable ping-timeout
[2015-06-06 04:23:57.902425] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
[2015-06-06 04:23:57.902546] D [socket.c:3816:socket_init] 0-snapd: disabling nodelay
[2015-06-06 04:23:57.902563] D [socket.c:3867:socket_init] 0-snapd: Configued transport.tcp-user-timeout=0
[2015-06-06 04:23:57.902579] D [socket.c:3938:socket_init] 0-snapd: SSL support on the I/O path is NOT enabled
[2015-06-06 04:23:57.902591] D [socket.c:3941:socket_init] 0-snapd: SSL support for glusterd is NOT enabled
[2015-06-06 04:23:57.902604] D [socket.c:3958:socket_init] 0-snapd: using system polling thread
[2015-06-06 04:23:57.902651] D [glusterd-snapd-svc.c:112:glusterd_snapdsvc_init] 0-management: Returning 0
[2015-06-06 04:23:57.908407] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:57.908438] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:57.908476] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:57.908517] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.908533] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = type value = 2
[2015-06-06 04:23:57.908554] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.908567] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = count value = 2
[2015-06-06 04:23:57.908591] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.908604] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = status value = 1
[2015-06-06 04:23:57.908622] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.908635] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = sub_count value = 2
[2015-06-06 04:23:57.908654] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.908666] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1
[2015-06-06 04:23:57.908685] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.908697] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = replica_count value = 2
[2015-06-06 04:23:57.908716] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.908728] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = disperse_count value = 0
[2015-06-06 04:23:57.908748] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.908760] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = redundancy_count value = 0
[2015-06-06 04:23:57.908779] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.908792] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = version value = 25
[2015-06-06 04:23:57.908810] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.908823] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = transport-type value = 0
[2015-06-06 04:23:57.908842] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.908854] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = volume-id value = 7f7f5445-2622-4681-b2dd-2325f93c2549
[2015-06-06 04:23:57.908877] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.908890] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = username value = 297a4286-b7c7-4377-a9c5-df7e928c5231
[2015-06-06 04:23:57.908915] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.908928] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = password value = c72abad3-b7d6-45b0-b171-7c69e8d6ee1f
[2015-06-06 04:23:57.909003] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909018] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = op-version value = 30600
[2015-06-06 04:23:57.909043] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909056] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2
[2015-06-06 04:23:57.909076] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909089] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = parent_volname value = N/A
[2015-06-06 04:23:57.909113] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909125] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = restored_from_snap value = 00000000-0000-0000-0000-000000000000
[2015-06-06 04:23:57.909149] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909162] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = snap-max-hard-limit value = 256
[2015-06-06 04:23:57.909183] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909195] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = auth.ssl-allow value = *
[2015-06-06 04:23:57.909225] D [glusterd-store.c:2634:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=auth.ssl-allow,value:*
[2015-06-06 04:23:57.909249] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909262] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = client.ssl value = on
[2015-06-06 04:23:57.909282] D [glusterd-store.c:2634:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=client.ssl,value:on
[2015-06-06 04:23:57.909303] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909316] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = server.ssl value = on
[2015-06-06 04:23:57.909337] D [glusterd-store.c:2634:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=server.ssl,value:on
[2015-06-06 04:23:57.909357] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909370] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = config.transport value = tcp
[2015-06-06 04:23:57.909400] D [glusterd-store.c:2634:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=config.transport,value:tcp
[2015-06-06 04:23:57.909422] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909435] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-0 value = svr1:-home1
[2015-06-06 04:23:57.909457] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909470] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-1 value = svr2:-home1
[2015-06-06 04:23:57.909493] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:57.909542] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:57.909573] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:57.909609] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909634] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909651] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909669] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909686] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909703] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909720] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909737] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909754] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909771] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909806] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909823] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909840] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909856] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909873] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909891] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909908] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909926] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909965] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.909983] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.910000] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.910018] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.916756] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:57.916786] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:57.916827] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:57.916866] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.916887] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:57.916906] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.138315] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.138336] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.138346] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.138360] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.138372] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.138398] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.138413] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.146754] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.146785] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.146823] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.146861] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.146885] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.146898] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.146912] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.146925] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.146957] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.146972] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.146985] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.146999] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.147024] D [glusterd-store.c:2325:glusterd_store_retrieve_bricks] 0-: Returning with 0
[2015-06-06 04:23:58.147300] D [glusterd-utils.c:1964:glusterd_volume_compute_cksum] 0-management: Returning with 0
[2015-06-06 04:23:58.172095] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.172169] D [store.c:304:gf_store_retrieve_value] 0-: key version found
[2015-06-06 04:23:58.172207] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
[2015-06-06 04:23:58.188751] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.188866] D [glusterd-utils.c:1964:glusterd_volume_compute_cksum] 0-management: Returning with 0
[2015-06-06 04:23:58.205659] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.205780] D [store.c:372:gf_store_save_value] 0-: returning: 0
[2015-06-06 04:23:58.205829] D [store.c:372:gf_store_save_value] 0-: returning: 0
[2015-06-06 04:23:58.251394] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
[2015-06-06 04:23:58.259736] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.259766] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.259799] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.259836] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.259858] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.259876] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.259894] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.259915] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.259999] D [glusterd-utils.c:519:glusterd_volinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.260082] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2015-06-06 04:23:58.260100] D [rpc-clnt.c:989:rpc_clnt_connection_init] 0-snapd: disable ping-timeout
[2015-06-06 04:23:58.260122] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
[2015-06-06 04:23:58.260252] D [socket.c:3816:socket_init] 0-snapd: disabling nodelay
[2015-06-06 04:23:58.260270] D [socket.c:3867:socket_init] 0-snapd: Configued transport.tcp-user-timeout=0
[2015-06-06 04:23:58.260287] D [socket.c:3938:socket_init] 0-snapd: SSL support on the I/O path is NOT enabled
[2015-06-06 04:23:58.260300] D [socket.c:3941:socket_init] 0-snapd: SSL support for glusterd is NOT enabled
[2015-06-06 04:23:58.260313] D [socket.c:3958:socket_init] 0-snapd: using system polling thread
[2015-06-06 04:23:58.260358] D [glusterd-snapd-svc.c:112:glusterd_snapdsvc_init] 0-management: Returning 0
[2015-06-06 04:23:58.268158] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.268189] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.268226] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.268264] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268279] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = type value = 2
[2015-06-06 04:23:58.268300] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268314] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = count value = 14
[2015-06-06 04:23:58.268333] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268346] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = status value = 1
[2015-06-06 04:23:58.268364] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268377] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = sub_count value = 2
[2015-06-06 04:23:58.268396] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268409] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = stripe_count value = 1
[2015-06-06 04:23:58.268428] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268441] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = replica_count value = 2
[2015-06-06 04:23:58.268460] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268473] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = disperse_count value = 0
[2015-06-06 04:23:58.268492] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268505] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = redundancy_count value = 0
[2015-06-06 04:23:58.268524] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268537] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = version value = 52
[2015-06-06 04:23:58.268555] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268568] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = transport-type value = 0
[2015-06-06 04:23:58.268588] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268601] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = volume-id value = 6cbe45f3-361c-42a7-9f02-a7bb7e717cca
[2015-06-06 04:23:58.268624] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268637] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = username value = ee88b27e-ac6e-4109-8426-0027d9292ac9
[2015-06-06 04:23:58.268658] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268670] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = password value = 266e545f-ad38-47b2-9659-0acfd38c6a57
[2015-06-06 04:23:58.268692] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268705] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = op-version value = 30600
[2015-06-06 04:23:58.268726] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268739] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = client-op-version value = 2
[2015-06-06 04:23:58.268759] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268772] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = parent_volname value = N/A
[2015-06-06 04:23:58.268793] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268807] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = restored_from_snap value = 00000000-0000-0000-0000-000000000000
[2015-06-06 04:23:58.268830] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268843] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = snap-max-hard-limit value = 256
[2015-06-06 04:23:58.268864] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268877] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = config.transport value = tcp
[2015-06-06 04:23:58.268905] D [glusterd-store.c:2634:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=config.transport,value:tcp
[2015-06-06 04:23:58.268928] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.268979] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = client.ssl value = on
[2015-06-06 04:23:58.269002] D [glusterd-store.c:2634:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=client.ssl,value:on
[2015-06-06 04:23:58.269025] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269038] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = server.ssl value = on
[2015-06-06 04:23:58.269058] D [glusterd-store.c:2634:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=server.ssl,value:on
[2015-06-06 04:23:58.269078] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269091] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = auth.ssl-allow value = *
[2015-06-06 04:23:58.269111] D [glusterd-store.c:2634:glusterd_store_update_volinfo] 0-: Parsed as Volume-set:key=auth.ssl-allow,value:*
[2015-06-06 04:23:58.269131] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269144] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-0 value = svr1:-data2
[2015-06-06 04:23:58.269165] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269178] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-1 value = svr2:-data2
[2015-06-06 04:23:58.269199] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269211] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-2 value = svr1:-data3
[2015-06-06 04:23:58.269232] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269244] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-3 value = svr2:-data3
[2015-06-06 04:23:58.269266] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269279] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-4 value = svr1:-data4
[2015-06-06 04:23:58.269300] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269313] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-5 value = svr2:-data4
[2015-06-06 04:23:58.269334] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269346] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-6 value = svr1:-data5
[2015-06-06 04:23:58.269367] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269379] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-7 value = svr2:-data5
[2015-06-06 04:23:58.269400] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269413] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-8 value = svr1:-data1
[2015-06-06 04:23:58.269433] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269446] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-9 value = svr2:-data1
[2015-06-06 04:23:58.269466] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269479] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-10 value = svr1:-data6
[2015-06-06 04:23:58.269499] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269513] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-11 value = svr2:-data6
[2015-06-06 04:23:58.269533] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269546] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-12 value = svr1:-data7
[2015-06-06 04:23:58.269567] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269580] D [glusterd-store.c:2485:glusterd_store_update_volinfo] 0-: key = brick-13 value = svr2:-data7
[2015-06-06 04:23:58.269602] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.269647] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.269683] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.269716] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269736] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269753] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269771] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269789] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269806] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269823] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269840] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269858] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269877] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269895] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269913] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269930] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269972] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.269990] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.270007] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.270025] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.270043] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.270061] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.270078] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.270096] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.270114] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.270131] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.276498] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.276529] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.276567] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.276604] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.276626] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.276645] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.276665] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.276684] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.276703] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.276722] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.276741] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.276783] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.276809] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.284836] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.284867] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.284906] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.284966] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.284992] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.285012] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.285032] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.285051] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.285071] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.285090] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.285108] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.285129] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.285169] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.285197] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.293252] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.293283] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.293321] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.293357] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.293378] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.293397] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.293416] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.293435] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.293453] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.293471] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.293489] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.293529] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.293557] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.301526] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.301556] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.301597] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.301636] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.301658] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.301677] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.301696] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.301715] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.301733] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.301751] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.301769] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.301788] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.301828] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.301857] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.309975] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.310005] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.310044] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.310081] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.310102] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.310121] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.310141] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.310159] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.310177] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.310196] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.310214] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.310255] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.310280] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.318319] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.318349] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.318391] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.318431] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.318453] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.318472] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.318492] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.318511] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.318530] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.318549] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.318568] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.318587] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.318634] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.318661] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.326720] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.326751] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.326789] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.326826] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.326848] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.326867] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.326888] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.326911] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.326931] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.326981] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.327001] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.327042] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.327069] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.335084] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.335115] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.335153] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.335190] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.335212] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.335231] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.335251] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.335270] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.335289] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.335308] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.335327] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.335346] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.335388] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.335414] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.343459] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.343490] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.343529] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.343565] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.343587] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.343607] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.343627] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.343646] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.343665] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.343684] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.343704] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.343746] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.343771] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.351836] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.351867] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.351905] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.351972] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.351998] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.352017] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.352037] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.352056] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.352075] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.352093] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.352112] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.352131] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.352175] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.352201] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.360201] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.360232] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.360270] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.360306] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.360328] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.360347] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.360367] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.360386] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.360404] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.360422] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.360440] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.360459] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.360501] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.360527] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.368562] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.368592] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.368630] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.368667] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.368689] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.368708] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.368727] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.368745] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.368763] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.368781] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.368800] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.368818] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.368863] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.368889] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.376950] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.376981] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.377023] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.377060] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.377082] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.377101] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.377121] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.377140] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.377159] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.377178] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.377197] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.377216] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.377258] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.377284] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.385334] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.385365] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.385404] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.385440] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.385462] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.385482] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.385502] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.385521] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.385540] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.385559] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.385577] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.385597] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.385632] D [glusterd-store.c:2325:glusterd_store_retrieve_bricks] 0-: Returning with 0
[2015-06-06 04:23:58.386052] D [glusterd-utils.c:1964:glusterd_volume_compute_cksum] 0-management: Returning with 0
[2015-06-06 04:23:58.407228] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.407292] D [store.c:304:gf_store_retrieve_value] 0-: key version found
[2015-06-06 04:23:58.407329] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
[2015-06-06 04:23:58.415605] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.415720] D [glusterd-utils.c:1964:glusterd_volume_compute_cksum] 0-management: Returning with 0
[2015-06-06 04:23:58.432504] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.432624] D [store.c:372:gf_store_save_value] 0-: returning: 0
[2015-06-06 04:23:58.432673] D [store.c:372:gf_store_save_value] 0-: returning: 0
[2015-06-06 04:23:58.478139] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
[2015-06-06 04:23:58.486471] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.486501] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.486537] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.486577] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.486598] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.486616] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.486635] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.486656] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.486689] D [glusterd-store.c:2968:glusterd_store_retrieve_volumes] 0-: Returning with 0
[2015-06-06 04:23:58.503287] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.503317] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.503349] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.503397] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.503419] D [glusterd-utils.c:5763:glusterd_sm_tr_log_init] 0-: returning 0
[2015-06-06 04:23:58.503444] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.503463] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.503480] D [glusterd-peer-utils.c:454:glusterd_peer_hostname_new] 0-glusterd: Returning 0
[2015-06-06 04:23:58.503501] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.503537] I [glusterd-handler.c:3387:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2015-06-06 04:23:58.503611] D [glusterd-handler.c:3220:glusterd_transport_inet_options_build] 0-glusterd: Returning 0
[2015-06-06 04:23:58.503645] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2015-06-06 04:23:58.503660] D [rpc-clnt.c:984:rpc_clnt_connection_init] 0-management: setting ping-timeout to 30
[2015-06-06 04:23:58.503678] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
[2015-06-06 04:23:58.503832] D [socket.c:3867:socket_init] 0-management: Configued transport.tcp-user-timeout=-1
[2015-06-06 04:23:58.503852] D [socket.c:3938:socket_init] 0-management: SSL support on the I/O path is NOT enabled
[2015-06-06 04:23:58.503865] D [socket.c:3941:socket_init] 0-management: SSL support for glusterd is NOT enabled
[2015-06-06 04:23:58.503878] D [socket.c:3958:socket_init] 0-management: using system polling thread
[2015-06-06 04:23:58.510217] D [common-utils.c:334:gf_resolve_ip6] 0-resolver: returning ip-160.91.108.93 (port-24007) for hostname: svr2 and port: 24007
[2015-06-06 04:23:58.510246] D [socket.c:2883:socket_fix_ssl_opts] 0-management: disabling SSL for portmapper connection
[2015-06-06 04:23:58.510360] D [glusterd-handler.c:3153:glusterd_rpc_create] 0-management: returning 0
[2015-06-06 04:23:58.510387] D [glusterd-store.c:3986:glusterd_store_retrieve_peers] 0-: Returning with 0
[2015-06-06 04:23:58.510441] D [glusterd-store.c:3386:glusterd_store_retrieve_missed_snaps_list] 0-management: EOF for missed_snap_list
[2015-06-06 04:23:58.510462] D [glusterd-store.c:3487:glusterd_store_retrieve_snaps] 0-: Returning with 0
[2015-06-06 04:23:58.510758] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.510857] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.511276] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.511304] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.511312] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.511321] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.511330] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.511337] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.511498] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.511555] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.511790] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.511803] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.511811] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.511820] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.511828] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.511835] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.512020] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.512079] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.512313] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.512327] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.512334] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.512343] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.512351] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.512358] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.512514] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.512570] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.512810] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.512824] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.512832] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.512840] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.512848] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.512855] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.513037] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.513095] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.513329] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.513342] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.513350] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.513358] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.513366] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.513373] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.513529] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.513585] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.513818] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.513831] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.513839] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.513847] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.513855] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.513862] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.514044] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.514102] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.514337] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.514351] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.514359] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.514368] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.514375] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.514383] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.514540] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.514596] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.514835] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.514849] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.514856] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.514865] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.514873] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.514880] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.514890] D [glusterd-store.c:4196:glusterd_restore] 0-: Returning 0
[2015-06-06 04:23:58.533430] D [store.c:418:gf_store_handle_new] 0-: Returning 0
[2015-06-06 04:23:58.533460] D [store.c:437:gf_store_handle_retrieve] 0-: Returning 0
[2015-06-06 04:23:58.533492] D [store.c:499:gf_store_iter_new] 0-: Returning with 0
[2015-06-06 04:23:58.533529] D [store.c:608:gf_store_iter_get_next] 0-: Returning with 0
[2015-06-06 04:23:58.533552] D [store.c:608:gf_store_iter_get_next] 0-: Returning with -1
[2015-06-06 04:23:58.533576] D [store.c:458:gf_store_handle_destroy] 0-: Returning 0
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option transport.socket.listen-backlog 128
  7:     option ping-timeout 30
  8:     option transport.socket.read-fail-log off
  9:     option transport.socket.keepalive-interval 2
 10:     option transport.socket.keepalive-time 10
 11:     option transport-type rdma
 12:     option working-directory /var/lib/glusterd
 13: end-volume
 14:  
+------------------------------------------------------------------------------+
[2015-06-06 04:23:58.534013] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-06-06 04:23:58.534203] D [glusterd-peer-utils.c:977:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.534228] D [glusterd-handler.c:4793:__glusterd_peer_rpc_notify] 0-management: got RPC_CLNT_CONNECT
[2015-06-06 04:23:58.534252] D [glusterd-peer-utils.c:977:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.534935] D [rpc-clnt-ping.c:94:__rpc_clnt_remove_ping_timer] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7f8e23179f80] (--> /usr/lib64/libgfrpc.so.0(+0x1277c)[0x7f8e22f4d77c] (--> /usr/lib64/libgfrpc.so.0(+0x12a12)[0x7f8e22f4da12]
 (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x390)[0x7f8e22f4a210] (--> /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_submit_request_unlocked+0xd7)[0x7f8e18e99957] ))))) 0-: 160.91.108.93:24007: ping timer event already removed
[2015-06-06 04:23:58.535227] D [glusterd-peer-utils.c:977:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.535258] D [glusterd-handshake.c:2024:__glusterd_peer_dump_version_cbk] 0-management: Proceeding to op-version handshake with peer svr2
[2015-06-06 04:23:58.535296] D [glusterd-peer-utils.c:977:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.535847] D [glusterd-peer-utils.c:196:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.536032] D [glusterd-peer-utils.c:977:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.536091] D [glusterd-handshake.c:1602:gd_validate_peer_op_version] 0-management: Peer svr2 accepted
[2015-06-06 04:23:58.536476] I [glusterd-handshake.c:1151:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600
[2015-06-06 04:23:58.536707] D [store.c:372:gf_store_save_value] 0-: returning: 0
[2015-06-06 04:23:58.536759] D [store.c:372:gf_store_save_value] 0-: returning: 0
[2015-06-06 04:23:58.600270] D [glusterd-peer-utils.c:977:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.600315] D [glusterd-peer-utils.c:977:glusterd_peerinfo_find_by_generation] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.600334] D [glusterd-sm.c:1104:glusterd_friend_sm_inject_event] 0-glusterd: Enqueue event: 'GD_FRIEND_EVENT_CONNECTED'
[2015-06-06 04:23:58.600350] D [glusterd-handshake.c:1555:glusterd_event_connected_inject] 0-: returning 0
[2015-06-06 04:23:58.600366] D [glusterd-peer-utils.c:196:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.600381] D [glusterd-sm.c:1195:glusterd_friend_sm] 0-: Dequeued event of type: 'GD_FRIEND_EVENT_CONNECTED'
[2015-06-06 04:23:58.600396] D [glusterd-peer-utils.c:196:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.600413] D [glusterd-peer-utils.c:196:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.600944] D [glusterd-utils.c:2409:glusterd_add_volume_to_dict] 0-management: Returning with 0
[2015-06-06 04:23:58.601295] D [glusterd-utils.c:2409:glusterd_add_volume_to_dict] 0-management: Returning with 0
[2015-06-06 04:23:58.602191] D [glusterd-rpc-ops.c:1511:glusterd_rpc_friend_add] 0-glusterd: Returning 0
[2015-06-06 04:23:58.602208] D [glusterd-sm.c:321:glusterd_ac_friend_add] 0-: Returning with 0
[2015-06-06 04:23:58.602217] D [glusterd-peer-utils.c:196:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.602226] D [glusterd-utils.c:5812:glusterd_sm_tr_log_transition_add] 0-management: Transitioning from 'Peer in Cluster' to 'Peer in Cluster' due to event 'GD_FRIEND_EVENT_CONNECTED'
[2015-06-06 04:23:58.602237] D [glusterd-utils.c:5814:glusterd_sm_tr_log_transition_add] 0-management: returning 0
[2015-06-06 04:23:58.602245] D [glusterd-peer-utils.c:196:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.602260] D [glusterd-store.c:3673:glusterd_store_create_peer_dir] 0-: Returning with 0
[2015-06-06 04:23:58.602336] D [store.c:372:gf_store_save_value] 0-: returning: 0
[2015-06-06 04:23:58.602365] D [store.c:372:gf_store_save_value] 0-: returning: 0
[2015-06-06 04:23:58.602390] D [store.c:372:gf_store_save_value] 0-: returning: 0
[2015-06-06 04:23:58.602399] D [glusterd-store.c:3802:glusterd_store_peer_write] 0-: Returning with 0
[2015-06-06 04:23:58.646514] D [glusterd-store.c:3828:glusterd_store_perform_peer_store] 0-: Returning 0
[2015-06-06 04:23:58.646545] D [glusterd-store.c:3849:glusterd_store_peerinfo] 0-: Returning with 0
[2015-06-06 04:23:58.646628] I [glusterd-handler.c:2424:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 4b025ff9-597a-43d4-96f2-17ccf90dc7cf
[2015-06-06 04:23:58.646661] D [glusterd-peer-utils.c:196:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.646863] D [glusterd-sm.c:1104:glusterd_friend_sm_inject_event] 0-glusterd: Enqueue event: 'GD_FRIEND_EVENT_RCVD_FRIEND_REQ'
[2015-06-06 04:23:58.646880] D [glusterd-peer-utils.c:196:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.646894] D [glusterd-sm.c:1195:glusterd_friend_sm] 0-: Dequeued event of type: 'GD_FRIEND_EVENT_RCVD_FRIEND_REQ'
[2015-06-06 04:23:58.646909] D [glusterd-peer-utils.c:196:glusterd_peerinfo_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
[2015-06-06 04:23:58.646985] D [glusterd-utils.c:2733:import_prdict_dict] 0-: Returning with 0
[2015-06-06 04:23:58.647017] D [glusterd-utils.c:1347:glusterd_volinfo_find] 0-management: Volume data found
[2015-06-06 04:23:58.647031] D [glusterd-utils.c:1354:glusterd_volinfo_find] 0-management: Returning 0
[2015-06-06 04:23:58.647048] I [MSGID: 106009] [glusterd-utils.c:2600:glusterd_compare_friend_volume] 0-management: Version of volume data differ. local version = 52, remote version = 55 on peer svr2
[2015-06-06 04:23:58.647070] D [glusterd-utils.c:2680:glusterd_compare_friend_volume] 0-management: Returning with ret: 0, status: 2
[2015-06-06 04:23:58.647087] D [glusterd-utils.c:1347:glusterd_volinfo_find] 0-management: Volume home found
[2015-06-06 04:23:58.647105] D [glusterd-utils.c:1354:glusterd_volinfo_find] 0-management: Returning 0
[2015-06-06 04:23:58.647124] I [MSGID: 106009] [glusterd-utils.c:2600:glusterd_compare_friend_volume] 0-management: Version of volume home differ. local version = 25, remote version = 28 on peer svr2
[2015-06-06 04:23:58.647142] D [glusterd-utils.c:2680:glusterd_compare_friend_volume] 0-management: Returning with ret: 0, status: 2
[2015-06-06 04:23:58.647199] D [glusterd-utils.c:519:glusterd_volinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.647253] D [glusterd-utils.c:3095:gd_import_friend_volume_rebal_dict] 0-management: Returning with 0
[2015-06-06 04:23:58.647296] D [glusterd-utils.c:2733:import_prdict_dict] 0-: Returning with 0
[2015-06-06 04:23:58.647312] D [glusterd-utils.c:2733:import_prdict_dict] 0-: Returning with 0
[2015-06-06 04:23:58.647325] D [glusterd-utils.c:2811:glusterd_import_friend_volume_opts] 0-: Returning with 0
[2015-06-06 04:23:58.647373] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.647701] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.647795] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.648132] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.648156] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.648170] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.648184] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.648226] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.648254] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.648269] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.648282] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.648295] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.648329] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.648610] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.648702] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.649061] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.649085] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.649099] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.649112] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.649153] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.649181] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.649196] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.649209] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.649221] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.649261] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.649538] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.649630] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.649994] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.650018] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.650032] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.650045] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.650086] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.650115] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.650131] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.650143] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.650157] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.650203] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.650480] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.650571] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.650899] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.650922] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.650968] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.650983] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.651027] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.651058] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.651072] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.651085] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.651097] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.651132] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.651305] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.651363] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.651604] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.651619] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.651626] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.651633] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.651660] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.651679] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.651687] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.651695] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.651702] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.651728] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.651894] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.651969] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.652222] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.652237] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.652245] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.652252] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.652279] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.652298] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.652307] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.652314] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.652320] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.652350] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.652518] D [glusterd-peer-utils.c:129:glusterd_peerinfo_find_by_hostname] 0-management: Unable to find friend: svr1
[2015-06-06 04:23:58.652574] D [common-utils.c:3116:gf_is_local_addr] 0-management: 160.91.55.61 
[2015-06-06 04:23:58.652826] D [common-utils.c:3013:gf_interface_search] 0-management: 160.91.55.61 is local address at interface eth0
[2015-06-06 04:23:58.652840] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.652848] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.652855] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.652882] D [glusterd-utils.c:811:glusterd_brickinfo_new] 0-management: Returning 0
[2015-06-06 04:23:58.652901] D [glusterd-peer-utils.c:636:gd_peerinfo_find_from_hostname] 0-management: Friend svr2 found.. state: 3
[2015-06-06 04:23:58.652910] D [glusterd-peer-utils.c:164:glusterd_hostname_to_uuid] 0-management: returning 0
[2015-06-06 04:23:58.652917] D [glusterd-utils.c:851:glusterd_resolve_brick] 0-management: Returning 0
[2015-06-06 04:23:58.652924] D [glusterd-utils.c:2898:glusterd_import_new_brick] 0-: Returning with 0
[2015-06-06 04:23:58.652931] D [glusterd-utils.c:2938:glusterd_import_bricks] 0-: Returning with 0
[2015-06-06 04:23:58.652957] D [glusterd-utils.c:3477:glusterd_import_volinfo] 0-: Returning with 0
[2015-06-06 04:23:58.653022] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2015-06-06 04:23:58.653034] D [rpc-clnt.c:989:rpc_clnt_connection_init] 0-snapd: disable ping-timeout
[2015-06-06 04:23:58.653048] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
[2015-06-06 04:23:58.653128] D [socket.c:3816:socket_init] 0-snapd: disabling nodelay
[2015-06-06 04:23:58.653138] D [socket.c:3867:socket_init] 0-snapd: Configued transport.tcp-user-timeout=0
[2015-06-06 04:23:58.653147] D [socket.c:3938:socket_init] 0-snapd: SSL support on the I/O path is NOT enabled
[2015-06-06 04:23:58.653154] D [socket.c:3941:socket_init] 0-snapd: SSL support for glusterd is NOT enabled
[2015-06-06 04:23:58.653161] D [socket.c:3958:socket_init] 0-snapd: using system polling thread
[2015-06-06 04:23:58.653190] D [glusterd-snapd-svc.c:112:glusterd_snapdsvc_init] 0-management: Returning 0
[2015-06-06 04:23:58.653199] D [glusterd-utils.c:1347:glusterd_volinfo_find] 0-management: Volume data found
[2015-06-06 04:23:58.653206] D [glusterd-utils.c:1354:glusterd_volinfo_find] 0-management: Returning 0
[2015-06-06 04:23:58.653249] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2015-06-06 04:23:58.653263] D [rpc-clnt.c:989:rpc_clnt_connection_init] 0-snapd: disable ping-timeout
[2015-06-06 04:23:58.653273] D [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so
[2015-06-06 04:23:58.653343] D [socket.c:3816:socket_init] 0-snapd: disabling nodelay
[2015-06-06 04:23:58.653352] D [socket.c:3867:socket_init] 0-snapd: Configued transport.tcp-user-timeout=0
[2015-06-06 04:23:58.653361] D [socket.c:3938:socket_init] 0-snapd: SSL support on the I/O path is NOT enabled
[2015-06-06 04:23:58.653374] D [socket.c:3941:socket_init] 0-snapd: SSL support for glusterd is NOT enabled
[2015-06-06 04:23:58.653382] D [socket.c:3958:socket_init] 0-snapd: using system polling thread
[2015-06-06 04:23:58.653407] D [glusterd-snapd-svc.c:112:glusterd_snapdsvc_init] 0-management: Returning 0
*** buffer overflow detected ***: glusterd terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x3be9b02527]
/lib64/libc.so.6[0x3be9b00410]
/lib64/libc.so.6[0x3be9affb0b]
/lib64/libc.so.6(__snprintf_chk+0x7a)[0x3be9aff9da]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(gd_stop_rebalance_process+0xeb)[0x7f8e18e9e9ab]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(gd_check_and_update_rebalance_info+0x10f)[0x7f8e18e9ec7f]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_import_friend_volume+0xde)[0x7f8e18ea572e]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_import_friend_volumes+0x66)[0x7f8e18ea59f6]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_compare_friend_data+0x15d)[0x7f8e18ea5c2d]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(+0x539c3)[0x7f8e18e7f9c3]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_friend_sm+0x170)[0x7f8e18e806b0]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(__glusterd_handle_incoming_friend_req+0x232)[0x7f8e18e7eaa2]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x3f)[0x7f8e18e62e6f]
/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f8e22f44db5]
/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f8e22f44ff3]
/usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f8e22f468e8]
/usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so(+0x9bcd)[0x7f8e17c8fbcd]
/usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so(+0xb6fd)[0x7f8e17c916fd]
/usr/lib64/libglusterfs.so.0(+0x80f70)[0x7f8e231d5f70]
/lib64/libpthread.so.0[0x3be9e079d1]
/lib64/libc.so.6(clone+0x6d)[0x3be9ae88fd]
======= Memory map: ========
00400000-00412000 r-xp 00000000 103:02 658475                            /usr/sbin/glusterfsd
00612000-00614000 rw-p 00012000 103:02 658475                            /usr/sbin/glusterfsd
00db7000-00ecf000 rw-p 00000000 00:00 0                                  [heap]
3be9600000-3be9620000 r-xp 00000000 103:02 524290                        /lib64/ld-2.12.so
3be981f000-3be9820000 r--p 0001f000 103:02 524290                        /lib64/ld-2.12.so
3be9820000-3be9821000 rw-p 00020000 103:02 524290                        /lib64/ld-2.12.so
3be9821000-3be9822000 rw-p 00000000 00:00 0 
3be9a00000-3be9b8a000 r-xp 00000000 103:02 524291                        /lib64/libc-2.12.so
3be9b8a000-3be9d8a000 ---p 0018a000 103:02 524291                        /lib64/libc-2.12.so
3be9d8a000-3be9d8e000 r--p 0018a000 103:02 524291                        /lib64/libc-2.12.so
3be9d8e000-3be9d8f000 rw-p 0018e000 103:02 524291                        /lib64/libc-2.12.so
3be9d8f000-3be9d94000 rw-p 00000000 00:00 0 
3be9e00000-3be9e17000 r-xp 00000000 103:02 524294                        /lib64/libpthread-2.12.so
3be9e17000-3bea017000 ---p 00017000 103:02 524294                        /lib64/libpthread-2.12.so
3bea017000-3bea018000 r--p 00017000 103:02 524294                        /lib64/libpthread-2.12.so
3bea018000-3bea019000 rw-p 00018000 103:02 524294                        /lib64/libpthread-2.12.so
3bea019000-3bea01d000 rw-p 00000000 00:00 0 
3bea200000-3bea202000 r-xp 00000000 103:02 524304                        /lib64/libdl-2.12.so
3bea202000-3bea402000 ---p 00002000 103:02 524304                        /lib64/libdl-2.12.so
3bea402000-3bea403000 r--p 00002000 103:02 524304                        /lib64/libdl-2.12.so
3bea403000-3bea404000 rw-p 00003000 103:02 524304                        /lib64/libdl-2.12.so
3bea600000-3bea607000 r-xp 00000000 103:02 524314                        /lib64/librt-2.12.so
3bea607000-3bea806000 ---p 00007000 103:02 524314                        /lib64/librt-2.12.so
3bea806000-3bea807000 r--p 00006000 103:02 524314                        /lib64/librt-2.12.so
3bea807000-3bea808000 rw-p 00007000 103:02 524314                        /lib64/librt-2.12.so
3beaa00000-3beaa15000 r-xp 00000000 103:02 524303                        /lib64/libz.so.1.2.3
3beaa15000-3beac14000 ---p 00015000 103:02 524303                        /lib64/libz.so.1.2.3
3beac14000-3beac15000 r--p 00014000 103:02 524303                        /lib64/libz.so.1.2.3
3beac15000-3beac16000 rw-p 00015000 103:02 524303                        /lib64/libz.so.1.2.3
3beae00000-3beae83000 r-xp 00000000 103:02 524311                        /lib64/libm-2.12.so
3beae83000-3beb082000 ---p 00083000 103:02 524311                        /lib64/libm-2.12.so
3beb082000-3beb083000 r--p 00082000 103:02 524311                        /lib64/libm-2.12.so
3beb083000-3beb084000 rw-p 00083000 103:02 524311                        /lib64/libm-2.12.so
3beb200000-3beb243000 r-xp 00000000 103:02 524385                        /lib64/libdevmapper.so.1.02
3beb243000-3beb442000 ---p 00043000 103:02 524385                        /lib64/libdevmapper.so.1.02
3beb442000-3beb443000 r--p 00042000 103:02 524385                        /lib64/libdevmapper.so.1.02
3beb443000-3beb446000 rw-p 00043000 103:02 524385                        /lib64/libdevmapper.so.1.02
3beb446000-3beb447000 rw-p 00000000 00:00 0 
3beb600000-3beb61d000 r-xp 00000000 103:02 524333                        /lib64/libselinux.so.1
3beb61d000-3beb81c000 ---p 0001d000 103:02 524333                        /lib64/libselinux.so.1
3beb81c000-3beb81d000 r--p 0001c000 103:02 524333                        /lib64/libselinux.so.1
3beb81d000-3beb81e000 rw-p 0001d000 103:02 524333                        /lib64/libselinux.so.1
3beb81e000-3beb81f000 rw-p 00000000 00:00 0 
3beba00000-3beba16000 r-xp 00000000 103:02 524332                        /lib64/libresolv-2.12.so
3beba16000-3bebc16000 ---p 00016000 103:02 524332                        /lib64/libresolv-2.12.so
3bebc16000-3bebc17000 r--p 00016000 103:02 524332                        /lib64/libresolv-2.12.so
3bebc17000-3bebc18000 rw-p 00017000 103:02 524332                        /lib64/libresolv-2.12.so
3bebc18000-3bebc1a000 rw-p 00000000 00:00 0 
3bebe00000-3bebe16000 r-xp 00000000 103:02 524313                        /lib64/libgcc_s-4.4.7-20120601.so.1
3bebe16000-3bec015000 ---p 00016000 103:02 524313                        /lib64/libgcc_s-4.4.7-20120601.so.1
3bec015000-3bec016000 rw-p 00015000 103:02 524313                        /lib64/libgcc_s-4.4.7-20120601.so.1
3bec200000-3bec204000 r-xp 00000000 103:02 524375                        /lib64/libuuid.so.1.3.0
3bec204000-3bec403000 ---p 00004000 103:02 524375                        /lib64/libuuid.so.1.3.0
3bec403000-3bec404000 rw-p 00003000 103:02 524375                        /lib64/libuuid.so.1.3.0
3bee600000-3bee7b9000 r-xp 00000000 103:02 659921                        /usr/lib64/libcrypto.so.1.0.1e
3bee7b9000-3bee9b8000 ---p 001b9000 103:02 659921                        /usr/lib64/libcrypto.so.1.0.1e
3bee9b8000-3bee9d3000 r--p 001b8000 103:02 659921                        /usr/lib64/libcrypto.so.1.0.1e
3bee9d3000-3bee9df000 rw-p 001d3000 103:02 659921                        /usr/lib64/libcrypto.so.1.0.1e
3bee9df000-3bee9e3000 rw-p 00000000 00:00 0 
3bef200000-3bef203000 r-xp 00000000 103:02 524624                        /lib64/libcom_err.so.2.1
3bef203000-3bef402000 ---p 00003000 103:02 524624                        /lib64/libcom_err.so.2.1
3bef402000-3bef403000 r--p 00002000 103:02 524624                        /lib64/libcom_err.so.2.1
3bef403000-3bef404000 rw-p 00003000 103:02 524624                        /lib64/libcom_err.so.2.1
3befa00000-3befa29000 r-xp 00000000 103:02 524623                        /lib64/libk5crypto.so.3.1
3befa29000-3befc29000 ---p 00029000 103:02 524623                        /lib64/libk5crypto.so.3.1
3befc29000-3befc2a000 r--p 00029000 103:02 524623                        /lib64/libk5crypto.so.3.1
3befc2a000-3befc2b000 rw-p 0002a000 103:02 524623                        /lib64/libk5crypto.so.3.1
3befc2b000-3befc2c000 rw-p 00000000 00:00 0 
3befe00000-3befe02000 r-xp 00000000 103:02 524490                        /lib64/libkeyutils.so.1.3
3befe02000-3bf0001000 ---p 00002000 103:02 524490                        /lib64/libkeyutils.so.1.3[2015-06-06 04:23:58.654362] D [logging.c:1765:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to
 flush 5 extra log messages
The message "D [MSGID: 0] [glusterd-svc-mgmt.c:116:glusterd_svc_init_common] 0-management: Returning 0" repeated 4 times between [2015-06-06 04:23:57.900817] and [2015-06-06 04:23:57.902220]
[2015-06-06 04:23:58.655236] D [logging.c:1768:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 5 extra log messages
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash: 
2015-06-06 04:23:58
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7.1
/usr/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb6)[0x7f8e23179b66]
/usr/lib64/libglusterfs.so.0(gf_print_trace+0x33f)[0x7f8e2319859f]
/lib64/libc.so.6[0x3be9a326a0]
/lib64/libc.so.6(gsignal+0x35)[0x3be9a32625]
/lib64/libc.so.6(abort+0x175)[0x3be9a33e05]
/lib64/libc.so.6[0x3be9a70537]
/lib64/libc.so.6(__fortify_fail+0x37)[0x3be9b02527]
/lib64/libc.so.6[0x3be9b00410]
/lib64/libc.so.6[0x3be9affb0b]
/lib64/libc.so.6(__snprintf_chk+0x7a)[0x3be9aff9da]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(gd_stop_rebalance_process+0xeb)[0x7f8e18e9e9ab]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(gd_check_and_update_rebalance_info+0x10f)[0x7f8e18e9ec7f]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_import_friend_volume+0xde)[0x7f8e18ea572e]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_import_friend_volumes+0x66)[0x7f8e18ea59f6]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_compare_friend_data+0x15d)[0x7f8e18ea5c2d]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(+0x539c3)[0x7f8e18e7f9c3]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_friend_sm+0x170)[0x7f8e18e806b0]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(__glusterd_handle_incoming_friend_req+0x232)[0x7f8e18e7eaa2]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x3f)[0x7f8e18e62e6f]
/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x295)[0x7f8e22f44db5]
/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103)[0x7f8e22f44ff3]
/usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f8e22f468e8]
/usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so(+0x9bcd)[0x7f8e17c8fbcd]
/usr/lib64/glusterfs/3.7.1/rpc-transport/socket.so(+0xb6fd)[0x7f8e17c916fd]
/usr/lib64/libglusterfs.so.0(+0x80f70)[0x7f8e231d5f70]
/lib64/libpthread.so.0[0x3be9e079d1]
/lib64/libc.so.6(clone+0x6d)[0x3be9ae88fd]
---------
Aborted (core dumped)

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://www.gluster.org/mailman/listinfo/gluster-users