Re: add-brick: failed: Commit failed

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Ravi,

Please see the log attached. The output of "gluster volume status" is as follows. Should there be something listening on gfs3? I'm not sure whether it having TCP Port and Pid as N/A is a symptom or cause. Thank you.

# gluster volume status
Status of volume: gvol0
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick gfs1:/nodirectwritedata/gluster/gvol0 49152     0          Y       7706
Brick gfs2:/nodirectwritedata/gluster/gvol0 49152     0          Y       7624
Brick gfs3:/nodirectwritedata/gluster/gvol0 N/A       N/A        N       N/A 
Self-heal Daemon on localhost               N/A       N/A        Y       19853
Self-heal Daemon on gfs1                    N/A       N/A        Y       28600
Self-heal Daemon on gfs2                    N/A       N/A        Y       17614
 
Task Status of Volume gvol0
------------------------------------------------------------------------------
There are no active volume tasks


On Wed, 22 May 2019 at 18:06, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:

If you are trying this again, please 'gluster volume set $volname client-log-level DEBUG`before attempting the add-brick and attach the gvol0-add-brick-mount.log here. After that, you can change the client-log-level back to INFO.

-Ravi

On 22/05/19 11:32 AM, Ravishankar N wrote:


On 22/05/19 11:23 AM, David Cunningham wrote:
Hi Ravi,

I'd already done exactly that before, where step 3 was a simple 'rm -rf /nodirectwritedata/gluster/gvol0'. Have you another suggestion on what the cleanup or reformat should be?
`rm -rf /nodirectwritedata/gluster/gvol0` does look okay to me David. Basically, '/nodirectwritedata/gluster/gvol0' must be empty and must not have any extended attributes set on it. Why fuse_first_lookup() is failing is a bit of a mystery to me at this point. :-(
Regards,
Ravi

Thank you.


On Wed, 22 May 2019 at 13:56, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:

Hmm, so the volume info seems to indicate that the add-brick was successful but the gfid xattr is missing on the new brick (as are the actual files, barring the .glusterfs folder, according to your previous mail).

Do you want to try removing and adding it again?

1. `gluster volume remove-brick gvol0 replica 2 gfs3:/nodirectwritedata/gluster/gvol0 force` from gfs1

2. Check that gluster volume info is now back to a 1x2 volume on all nodes and `gluster peer status` is  connected on all nodes.

3. Cleanup or reformat '/nodirectwritedata/gluster/gvol0' on gfs3.

4. `gluster volume add-brick gvol0 replica 3 arbiter 1 gfs3:/nodirectwritedata/gluster/gvol0` from gfs1.

5. Check that the files are getting healed on to the new brick.

Thanks,
Ravi
On 22/05/19 6:50 AM, David Cunningham wrote:
Hi Ravi,

Certainly. On the existing two nodes:

gfs1 # getfattr -d -m. -e hex /nodirectwritedata/gluster/gvol0
getfattr: Removing leading '/' from absolute path names
# file: nodirectwritedata/gluster/gvol0
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.gvol0-client-2=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.volume-id=0xfb5af69e1c3e41648b23c1d7bec9b1b6

gfs2 # getfattr -d -m. -e hex /nodirectwritedata/gluster/gvol0
getfattr: Removing leading '/' from absolute path names
# file: nodirectwritedata/gluster/gvol0
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.gvol0-client-0=0x000000000000000000000000
trusted.afr.gvol0-client-2=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.volume-id=0xfb5af69e1c3e41648b23c1d7bec9b1b6

On the new node:

gfs3 # getfattr -d -m. -e hex /nodirectwritedata/gluster/gvol0
getfattr: Removing leading '/' from absolute path names
# file: nodirectwritedata/gluster/gvol0
trusted.afr.dirty=0x000000000000000000000001
trusted.glusterfs.volume-id=0xfb5af69e1c3e41648b23c1d7bec9b1b6

Output of "gluster volume info" is the same on all 3 nodes and is:

# gluster volume info
 
Volume Name: gvol0
Type: Replicate
Volume ID: fb5af69e-1c3e-4164-8b23-c1d7bec9b1b6
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: gfs1:/nodirectwritedata/gluster/gvol0
Brick2: gfs2:/nodirectwritedata/gluster/gvol0
Brick3: gfs3:/nodirectwritedata/gluster/gvol0 (arbiter)
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet


On Wed, 22 May 2019 at 12:43, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:
Hi David,
Could you provide the `getfattr -d -m. -e hex /nodirectwritedata/gluster/gvol0` output of all bricks and the output of `gluster volume info`?

Thanks,
Ravi
On 22/05/19 4:57 AM, David Cunningham wrote:
Hi Sanju,

Here's what glusterd.log says on the new arbiter server when trying to add the node:

[2019-05-22 00:15:05.963059] I [run.c:242:runner_log] (-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x3b2cd) [0x7fe4ca9102cd] -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0xe6b85) [0x7fe4ca9bbb85] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fe4d5ecc955] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh --volname=gvol0 --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd
[2019-05-22 00:15:05.963177] I [MSGID: 106578] [glusterd-brick-ops.c:1355:glusterd_op_perform_add_bricks] 0-management: replica-count is set 3
[2019-05-22 00:15:05.963228] I [MSGID: 106578] [glusterd-brick-ops.c:1360:glusterd_op_perform_add_bricks] 0-management: arbiter-count is set 1
[2019-05-22 00:15:05.963257] I [MSGID: 106578] [glusterd-brick-ops.c:1364:glusterd_op_perform_add_bricks] 0-management: type is set 0, need to change it
[2019-05-22 00:15:17.015268] E [MSGID: 106053] [glusterd-utils.c:13942:glusterd_handle_replicate_brick_ops] 0-management: Failed to set extended attribute trusted.add-brick : Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 00:15:17.036479] E [MSGID: 106073] [glusterd-brick-ops.c:2595:glusterd_op_add_brick] 0-glusterd: Unable to add bricks
[2019-05-22 00:15:17.036595] E [MSGID: 106122] [glusterd-mgmt.c:299:gd_mgmt_v3_commit_fn] 0-management: Add-brick commit failed.
[2019-05-22 00:15:17.036710] E [MSGID: 106122] [glusterd-mgmt-handler.c:594:glusterd_handle_commit_fn] 0-management: commit failed on operation Add brick

As before gvol0-add-brick-mount.log said:

[2019-05-22 00:15:17.005695] I [fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2019-05-22 00:15:17.005749] I [fuse-bridge.c:4878:fuse_graph_sync] 0-fuse: switched to graph 0
[2019-05-22 00:15:17.010101] E [fuse-bridge.c:4336:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
[2019-05-22 00:15:17.014217] W [fuse-bridge.c:897:fuse_attr_cbk] 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Transport endpoint is not connected)
[2019-05-22 00:15:17.015097] W [fuse-resolve.c:127:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)
[2019-05-22 00:15:17.015158] W [fuse-bridge.c:3294:fuse_setxattr_resume] 0-glusterfs-fuse: 3: SETXATTR 00000000-0000-0000-0000-000000000001/1 (trusted.add-brick) resolution failed
[2019-05-22 00:15:17.035636] I [fuse-bridge.c:5144:fuse_thread_proc] 0-fuse: initating unmount of /tmp/mntYGNbj9
[2019-05-22 00:15:17.035854] W [glusterfsd.c:1500:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f7745ccedd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55c81b63de75] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55c81b63dceb] ) 0-: received signum (15), shutting down
[2019-05-22 00:15:17.035942] I [fuse-bridge.c:5914:fini] 0-fuse: Unmounting '/tmp/mntYGNbj9'.
[2019-05-22 00:15:17.035966] I [fuse-bridge.c:5919:fini] 0-fuse: Closing fuse connection to '/tmp/mntYGNbj9'.

Here are the processes running on the new arbiter server:
# ps -ef | grep gluster
root      3466     1  0 20:13 ?        00:00:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/24c12b09f93eec8e.socket --xlator-option *replicate*.node-uuid=2069cfb3-c798-47e3-8cf8-3c584cf7c412 --process-name glustershd
root      6832     1  0 May16 ?        00:02:10 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
root     17841     1  0 May16 ?        00:00:58 /usr/sbin/glusterfs --process-name fuse --volfile-server=gfs1 --volfile-id=/gvol0 /mnt/glusterfs

Here are the files created on the new arbiter server:
# find /nodirectwritedata/gluster/gvol0 | xargs ls -ald
drwxr-xr-x 3 root root 4096 May 21 20:15 /nodirectwritedata/gluster/gvol0
drw------- 2 root root 4096 May 21 20:15 /nodirectwritedata/gluster/gvol0/.glusterfs

Thank you for your help!


On Tue, 21 May 2019 at 00:10, Sanju Rakonde <srakonde@xxxxxxxxxx> wrote:
David,

can you please attach glusterd.logs? As the error message says, Commit failed on the arbitar node, we might be able to find some issue on that node.

On Mon, May 20, 2019 at 10:10 AM Nithya Balachandran <nbalacha@xxxxxxxxxx> wrote:


On Fri, 17 May 2019 at 06:01, David Cunningham <dcunningham@xxxxxxxxxxxxx> wrote:
Hello,

We're adding an arbiter node to an existing volume and having an issue. Can anyone help? The root cause error appears to be "00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)", as below.

We are running glusterfs 5.6.1. Thanks in advance for any assistance!

On existing node gfs1, trying to add new arbiter node gfs3:

# gluster volume add-brick gvol0 replica 3 arbiter 1 gfs3:/nodirectwritedata/gluster/gvol0
volume add-brick: failed: Commit failed on gfs3. Please check log file for details.

This looks like a glusterd issue. Please check the glusterd logs for more info.
Adding the glusterd dev to this thread. Sanju, can you take a look?
 
Regards,
Nithya

On new node gfs3 in gvol0-add-brick-mount.log:

[2019-05-17 01:20:22.689721] I [fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2019-05-17 01:20:22.689778] I [fuse-bridge.c:4878:fuse_graph_sync] 0-fuse: switched to graph 0
[2019-05-17 01:20:22.694897] E [fuse-bridge.c:4336:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
[2019-05-17 01:20:22.699770] W [fuse-resolve.c:127:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)
[2019-05-17 01:20:22.699834] W [fuse-bridge.c:3294:fuse_setxattr_resume] 0-glusterfs-fuse: 2: SETXATTR 00000000-0000-0000-0000-000000000001/1 (trusted.add-brick) resolution failed
[2019-05-17 01:20:22.715656] I [fuse-bridge.c:5144:fuse_thread_proc] 0-fuse: initating unmount of /tmp/mntQAtu3f
[2019-05-17 01:20:22.715865] W [glusterfsd.c:1500:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7fb223bf6dd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x560886581e75] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x560886581ceb] ) 0-: received signum (15), shutting down
[2019-05-17 01:20:22.715926] I [fuse-bridge.c:5914:fini] 0-fuse: Unmounting '/tmp/mntQAtu3f'.
[2019-05-17 01:20:22.715953] I [fuse-bridge.c:5919:fini] 0-fuse: Closing fuse connection to '/tmp/mntQAtu3f'.

Processes running on new node gfs3:

# ps -ef | grep gluster
root      6832     1  0 20:17 ?        00:00:00 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
root     15799     1  0 20:17 ?        00:00:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/24c12b09f93eec8e.socket --xlator-option *replicate*.node-uuid=2069cfb3-c798-47e3-8cf8-3c584cf7c412 --process-name glustershd
root     16856 16735  0 21:21 pts/0    00:00:00 grep --color=auto gluster

--
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users


--
Thanks,
Sanju


--
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users


--
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782


--
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782


--
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782
[2019-05-22 23:03:23.344284] I [MSGID: 100030] [glusterfsd.c:2725:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 5.6 (args: /usr/sbin/glusterfs --volfile /tmp/gvol0.tcp-fuse.vol --client-pid -6 -l /var/log/glusterfs/gvol0-add-brick-mount.log /tmp/mntBBQr8A)
[2019-05-22 23:03:23.351566] D [MSGID: 0] [quick-read.c:1183:check_cache_size_ok] 0-gvol0-quick-read: Max cache size is 67334021120
[2019-05-22 23:03:23.351743] D [MSGID: 0] [io-cache.c:1595:check_cache_size_ok] 0-gvol0-io-cache: Max cache size is 67334021120
[2019-05-22 23:03:23.351809] D [MSGID: 0] [options.c:1227:xlator_option_init_size_uint64] 0-gvol0-readdir-ahead: option rda-request-size using set value 131072
[2019-05-22 23:03:23.351844] D [MSGID: 0] [options.c:1227:xlator_option_init_size_uint64] 0-gvol0-readdir-ahead: option rda-cache-limit using set value 10MB
[2019-05-22 23:03:23.351870] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-gvol0-readdir-ahead: option parallel-readdir using set value off
[2019-05-22 23:03:23.352098] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-gvol0-dht: option lock-migration using set value off
[2019-05-22 23:03:23.352143] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-gvol0-dht: option force-migration using set value off
[2019-05-22 23:03:23.352251] D [MSGID: 0] [dht-shared.c:356:dht_init_regex] 0-gvol0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
[2019-05-22 23:03:23.352376] D [MSGID: 0] [options.c:1224:xlator_option_init_uint32] 0-gvol0-replicate-0: option arbiter-count using set value 1
[2019-05-22 23:03:23.352810] D [MSGID: 0] [options.c:1221:xlator_option_init_str] 0-gvol0-replicate-0: option afr-pending-xattr using set value gvol0-client-0,gvol0-client-1,gvol0-client-2
[2019-05-22 23:03:23.354194] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-gvol0-client-2: option ping-timeout using set value 42
[2019-05-22 23:03:23.354231] D [MSGID: 0] [options.c:1232:xlator_option_init_path] 0-gvol0-client-2: option remote-subvolume using set value /nodirectwritedata/gluster/gvol0
[2019-05-22 23:03:23.354263] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-gvol0-client-2: option send-gids using set value true
[2019-05-22 23:03:23.354325] D [rpc-clnt.c:1002:rpc_clnt_connection_init] 0-gvol0-client-2: defaulting frame-timeout to 30mins
[2019-05-22 23:03:23.354352] D [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-gvol0-client-2: setting ping-timeout to 42
[2019-05-22 23:03:23.354387] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.6/rpc-transport/socket.so
[2019-05-22 23:03:23.361071] D [socket.c:4466:socket_init] 0-gvol0-client-2: Configued transport.tcp-user-timeout=0
[2019-05-22 23:03:23.361108] D [socket.c:4484:socket_init] 0-gvol0-client-2: Reconfigued transport.keepalivecnt=9
[2019-05-22 23:03:23.361129] D [socket.c:4169:ssl_setup_connection_params] 0-gvol0-client-2: SSL support on the I/O path is NOT enabled
[2019-05-22 23:03:23.361144] D [socket.c:4172:ssl_setup_connection_params] 0-gvol0-client-2: SSL support for glusterd is NOT enabled
[2019-05-22 23:03:23.361178] D [rpc-clnt.c:1579:rpcclnt_cbk_program_register] 0-gvol0-client-2: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
[2019-05-22 23:03:23.361200] D [MSGID: 0] [client.c:2547:client_init_rpc] 0-gvol0-client-2: client init successful
[2019-05-22 23:03:23.361271] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-gvol0-client-1: option ping-timeout using set value 42
[2019-05-22 23:03:23.361303] D [MSGID: 0] [options.c:1232:xlator_option_init_path] 0-gvol0-client-1: option remote-subvolume using set value /nodirectwritedata/gluster/gvol0
[2019-05-22 23:03:23.361336] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-gvol0-client-1: option send-gids using set value true
[2019-05-22 23:03:23.361357] D [rpc-clnt.c:1002:rpc_clnt_connection_init] 0-gvol0-client-1: defaulting frame-timeout to 30mins
[2019-05-22 23:03:23.361371] D [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-gvol0-client-1: setting ping-timeout to 42
[2019-05-22 23:03:23.361390] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.6/rpc-transport/socket.so
[2019-05-22 23:03:23.361564] D [socket.c:4466:socket_init] 0-gvol0-client-1: Configued transport.tcp-user-timeout=0
[2019-05-22 23:03:23.361591] D [socket.c:4484:socket_init] 0-gvol0-client-1: Reconfigued transport.keepalivecnt=9
[2019-05-22 23:03:23.361605] D [socket.c:4169:ssl_setup_connection_params] 0-gvol0-client-1: SSL support on the I/O path is NOT enabled
[2019-05-22 23:03:23.361617] D [socket.c:4172:ssl_setup_connection_params] 0-gvol0-client-1: SSL support for glusterd is NOT enabled
[2019-05-22 23:03:23.361630] D [rpc-clnt.c:1579:rpcclnt_cbk_program_register] 0-gvol0-client-1: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
[2019-05-22 23:03:23.361649] D [MSGID: 0] [client.c:2547:client_init_rpc] 0-gvol0-client-1: client init successful
[2019-05-22 23:03:23.361699] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-gvol0-client-0: option ping-timeout using set value 42
[2019-05-22 23:03:23.361734] D [MSGID: 0] [options.c:1232:xlator_option_init_path] 0-gvol0-client-0: option remote-subvolume using set value /nodirectwritedata/gluster/gvol0
[2019-05-22 23:03:23.361762] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-gvol0-client-0: option send-gids using set value true
[2019-05-22 23:03:23.361784] D [rpc-clnt.c:1002:rpc_clnt_connection_init] 0-gvol0-client-0: defaulting frame-timeout to 30mins
[2019-05-22 23:03:23.361797] D [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-gvol0-client-0: setting ping-timeout to 42
[2019-05-22 23:03:23.361829] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.6/rpc-transport/socket.so
[2019-05-22 23:03:23.361984] D [socket.c:4466:socket_init] 0-gvol0-client-0: Configued transport.tcp-user-timeout=0
[2019-05-22 23:03:23.362004] D [socket.c:4484:socket_init] 0-gvol0-client-0: Reconfigued transport.keepalivecnt=9
[2019-05-22 23:03:23.362018] D [socket.c:4169:ssl_setup_connection_params] 0-gvol0-client-0: SSL support on the I/O path is NOT enabled
[2019-05-22 23:03:23.362031] D [socket.c:4172:ssl_setup_connection_params] 0-gvol0-client-0: SSL support for glusterd is NOT enabled
[2019-05-22 23:03:23.362046] D [rpc-clnt.c:1579:rpcclnt_cbk_program_register] 0-gvol0-client-0: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
[2019-05-22 23:03:23.362060] D [MSGID: 0] [client.c:2547:client_init_rpc] 0-gvol0-client-0: client init successful
[2019-05-22 23:03:23.362168] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-2: option 'transport.address-family' is not recognized
[2019-05-22 23:03:23.362194] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-2: option 'transport.tcp-user-timeout' is not recognized
[2019-05-22 23:03:23.362218] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-2: option 'transport.socket.keepalive-time' is not recognized
[2019-05-22 23:03:23.362239] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-2: option 'transport.socket.keepalive-interval' is not recognized
[2019-05-22 23:03:23.362260] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-2: option 'transport.socket.keepalive-count' is not recognized
[2019-05-22 23:03:23.362293] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-1: option 'transport.address-family' is not recognized
[2019-05-22 23:03:23.362317] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-1: option 'transport.tcp-user-timeout' is not recognized
[2019-05-22 23:03:23.362338] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-1: option 'transport.socket.keepalive-time' is not recognized
[2019-05-22 23:03:23.362358] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-1: option 'transport.socket.keepalive-interval' is not recognized
[2019-05-22 23:03:23.362380] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-1: option 'transport.socket.keepalive-count' is not recognized
[2019-05-22 23:03:23.362427] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-0: option 'transport.address-family' is not recognized
[2019-05-22 23:03:23.362447] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-0: option 'transport.tcp-user-timeout' is not recognized
[2019-05-22 23:03:23.362468] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-0: option 'transport.socket.keepalive-time' is not recognized
[2019-05-22 23:03:23.362502] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-0: option 'transport.socket.keepalive-interval' is not recognized
[2019-05-22 23:03:23.362530] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-0: option 'transport.socket.keepalive-count' is not recognized
[2019-05-22 23:03:23.362561] D [fuse-bridge.c:5332:notify] 0-fuse: got event 12 on graph 0
[2019-05-22 23:03:23.362615] D [MSGID: 0] [afr-common.c:5012:__afr_launch_notify_timer] 0-gvol0-replicate-0: Initiating child-down timer
[2019-05-22 23:03:23.362652] I [MSGID: 114020] [client.c:2358:notify] 0-gvol0-client-0: parent translators are ready, attempting connect on transport
[2019-05-22 23:03:23.367816] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-69.42.167.137 (port-24007) for hostname: gfs1 and port: 24007
[2019-05-22 23:03:23.367850] D [socket.c:3223:socket_fix_ssl_opts] 0-gvol0-client-0: disabling SSL for portmapper connection
[2019-05-22 23:03:23.368114] I [MSGID: 114020] [client.c:2358:notify] 0-gvol0-client-1: parent translators are ready, attempting connect on transport
[2019-05-22 23:03:23.372391] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-69.42.172.137 (port-24007) for hostname: gfs2 and port: 24007
[2019-05-22 23:03:23.372421] D [socket.c:3223:socket_fix_ssl_opts] 0-gvol0-client-1: disabling SSL for portmapper connection
[2019-05-22 23:03:23.372535] I [MSGID: 114020] [client.c:2358:notify] 0-gvol0-client-2: parent translators are ready, attempting connect on transport
[2019-05-22 23:03:23.376795] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-192.157.88.220 (port-24007) for hostname: gfs3 and port: 24007
[2019-05-22 23:03:23.376825] D [socket.c:3223:socket_fix_ssl_opts] 0-gvol0-client-2: disabling SSL for portmapper connection
[2019-05-22 23:03:23.378012] I [MSGID: 101190] [event-epoll.c:621:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2019-05-22 23:03:23.378090] I [MSGID: 101190] [event-epoll.c:621:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2019-05-22 23:03:23.378143] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-0: got RPC_CLNT_CONNECT
[2019-05-22 23:03:23.378154] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-1: got RPC_CLNT_CONNECT
[2019-05-22 23:03:23.378862] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 69.42.172.137:24007: ping timer event already removed
[2019-05-22 23:03:23.378881] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 69.42.167.137:24007: ping timer event already removed
[2019-05-22 23:03:23.378971] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_CONNECT
[2019-05-22 23:03:23.379191] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 192.157.88.220:24007: ping timer event already removed
[2019-05-22 23:03:23.379300] D [MSGID: 0] [client-handshake.c:1392:server_has_portmap] 0-gvol0-client-2: detected portmapper on server
[2019-05-22 23:03:23.379306] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-2: Ping latency is 0ms
[2019-05-22 23:03:23.379445] E [MSGID: 114058] [client-handshake.c:1449:client_query_portmap_cbk] 0-gvol0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2019-05-22 23:03:23.379512] D [socket.c:2929:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:13) (non-SSL)
[2019-05-22 23:03:23.379544] D [MSGID: 0] [client.c:2275:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_DISCONNECT
[2019-05-22 23:03:23.379562] I [MSGID: 114018] [client.c:2288:client_rpc_notify] 0-gvol0-client-2: disconnected from gvol0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2019-05-22 23:03:23.379586] E [MSGID: 108006] [afr-common.c:5314:__afr_handle_child_down_event] 0-gvol0-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
[2019-05-22 23:03:23.379594] D [MSGID: 0] [client-handshake.c:1392:server_has_portmap] 0-gvol0-client-0: detected portmapper on server
[2019-05-22 23:03:23.380198] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-0: Ping latency is 1ms
[2019-05-22 23:03:23.380367] D [MSGID: 0] [client-handshake.c:1392:server_has_portmap] 0-gvol0-client-1: detected portmapper on server
[2019-05-22 23:03:23.380892] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-gvol0-client-0: changing port to 49152 (from 0)
[2019-05-22 23:03:23.380927] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-1: Ping latency is 2ms
[2019-05-22 23:03:23.380951] D [socket.c:2929:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:11) (non-SSL)
[2019-05-22 23:03:23.380974] D [MSGID: 0] [client.c:2275:client_rpc_notify] 0-gvol0-client-0: got RPC_CLNT_DISCONNECT
[2019-05-22 23:03:23.380992] D [MSGID: 0] [client.c:2316:client_rpc_notify] 0-gvol0-client-0: disconnected (skipped notify)
[2019-05-22 23:03:23.382318] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-gvol0-client-1: changing port to 49152 (from 0)
[2019-05-22 23:03:23.382377] D [socket.c:2929:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:12) (non-SSL)
[2019-05-22 23:03:23.382398] D [MSGID: 0] [client.c:2275:client_rpc_notify] 0-gvol0-client-1: got RPC_CLNT_DISCONNECT
[2019-05-22 23:03:23.382417] D [MSGID: 0] [client.c:2316:client_rpc_notify] 0-gvol0-client-1: disconnected (skipped notify)
[2019-05-22 23:03:23.384934] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-69.42.167.137 (port-24007) for hostname: gfs1 and port: 24007
[2019-05-22 23:03:23.386356] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-0: got RPC_CLNT_CONNECT
[2019-05-22 23:03:23.386603] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 69.42.167.137:49152: ping timer event already removed
[2019-05-22 23:03:23.387777] W [dict.c:1002:str_to_data] (-->/usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x403f7) [0x7f54b478d3f7] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f54bfe15b76] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f54bfe12491] ) 0-dict: value is NULL [Invalid argument]
[2019-05-22 23:03:23.387811] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-gvol0-client-0: failed to set process-name in handshake msg
[2019-05-22 23:03:23.387909] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-0: Ping latency is 1ms
[2019-05-22 23:03:23.388792] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-69.42.172.137 (port-24007) for hostname: gfs2 and port: 24007
[2019-05-22 23:03:23.389716] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-gvol0-client-0: Connected to gvol0-client-0, attached to remote volume '/nodirectwritedata/gluster/gvol0'.
[2019-05-22 23:03:23.389754] D [MSGID: 0] [client-handshake.c:945:client_post_handshake] 0-gvol0-client-0: No fds to open - notifying all parents child up
[2019-05-22 23:03:23.389780] D [MSGID: 0] [afr-common.c:5157:afr_get_halo_latency] 0-gvol0-replicate-0: Using halo latency 5
[2019-05-22 23:03:23.389811] I [MSGID: 108005] [afr-common.c:5237:__afr_handle_child_up_event] 0-gvol0-replicate-0: Subvolume 'gvol0-client-0' came back up; going online.
[2019-05-22 23:03:27.352904] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-192.157.88.220 (port-24007) for hostname: gfs3 and port: 24007
[2019-05-22 23:03:27.352959] D [socket.c:3223:socket_fix_ssl_opts] 0-gvol0-client-2: disabling SSL for portmapper connection
[2019-05-22 23:03:27.353228] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_CONNECT
[2019-05-22 23:03:27.353608] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 192.157.88.220:24007: ping timer event already removed
[2019-05-22 23:03:27.353741] D [MSGID: 0] [client-handshake.c:1392:server_has_portmap] 0-gvol0-client-2: detected portmapper on server
[2019-05-22 23:03:27.353853] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-2: Ping latency is 0ms
[2019-05-22 23:03:27.353922] D [MSGID: 0] [client-handshake.c:1455:client_query_portmap_cbk] 0-gvol0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2019-05-22 23:03:27.353997] D [socket.c:2929:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:11) (non-SSL)
[2019-05-22 23:03:27.354042] D [MSGID: 0] [client.c:2275:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_DISCONNECT
[2019-05-22 23:03:27.354073] D [MSGID: 0] [client.c:2296:client_rpc_notify] 0-gvol0-client-2: disconnected from gvol0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2019-05-22 23:03:30.359183] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-192.157.88.220 (port-24007) for hostname: gfs3 and port: 24007
[2019-05-22 23:03:30.359242] D [socket.c:3223:socket_fix_ssl_opts] 0-gvol0-client-2: disabling SSL for portmapper connection
[2019-05-22 23:03:30.359481] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_CONNECT
[2019-05-22 23:03:30.359846] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 192.157.88.220:24007: ping timer event already removed
[2019-05-22 23:03:30.359982] D [MSGID: 0] [client-handshake.c:1392:server_has_portmap] 0-gvol0-client-2: detected portmapper on server
[2019-05-22 23:03:30.360096] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-2: Ping latency is 0ms
[2019-05-22 23:03:30.360170] D [MSGID: 0] [client-handshake.c:1455:client_query_portmap_cbk] 0-gvol0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2019-05-22 23:03:30.360248] D [socket.c:2929:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:11) (non-SSL)
[2019-05-22 23:03:30.360298] D [MSGID: 0] [client.c:2275:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_DISCONNECT
[2019-05-22 23:03:30.360332] D [MSGID: 0] [client.c:2296:client_rpc_notify] 0-gvol0-client-2: disconnected from gvol0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2019-05-22 23:03:33.365443] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-192.157.88.220 (port-24007) for hostname: gfs3 and port: 24007
[2019-05-22 23:03:33.365507] D [socket.c:3223:socket_fix_ssl_opts] 0-gvol0-client-2: disabling SSL for portmapper connection
[2019-05-22 23:03:33.365780] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_CONNECT
[2019-05-22 23:03:33.366149] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 192.157.88.220:24007: ping timer event already removed
[2019-05-22 23:03:33.366275] D [MSGID: 0] [client-handshake.c:1392:server_has_portmap] 0-gvol0-client-2: detected portmapper on server
[2019-05-22 23:03:33.366332] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-2: Ping latency is 0ms
[2019-05-22 23:03:33.366537] D [MSGID: 0] [client-handshake.c:1455:client_query_portmap_cbk] 0-gvol0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2019-05-22 23:03:33.366625] D [socket.c:2929:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:11) (non-SSL)
[2019-05-22 23:03:33.366658] D [MSGID: 0] [client.c:2275:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_DISCONNECT
[2019-05-22 23:03:33.366687] D [MSGID: 0] [client.c:2296:client_rpc_notify] 0-gvol0-client-2: disconnected from gvol0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2019-05-22 23:03:34.365978] D [fuse-bridge.c:5332:notify] 0-fuse: got event 5 on graph 0
[2019-05-22 23:03:34.367664] D [MSGID: 0] [dht-diskusage.c:94:dht_du_info_cbk] 0-gvol0-dht: subvolume 'gvol0-replicate-0': avail_percent is: 82.00 and avail_space is: 1554380546048 and avail_inodes is: 98.00
[2019-05-22 23:03:34.367800] D [fuse-bridge.c:4919:fuse_get_mount_status] 0-fuse: mount status is 0
[2019-05-22 23:03:34.368012] D [fuse-bridge.c:4206:fuse_init] 0-glusterfs-fuse: Detected support for FUSE_AUTO_INVAL_DATA. Enabling fopen_keep_cache automatically.
[2019-05-22 23:03:34.368073] I [fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2019-05-22 23:03:34.368102] I [fuse-bridge.c:4878:fuse_graph_sync] 0-fuse: switched to graph 0
[2019-05-22 23:03:34.368323] D [MSGID: 0] [dht-common.c:3454:dht_do_fresh_lookup] 0-gvol0-dht: Calling fresh lookup for / on gvol0-replicate-0
[2019-05-22 23:03:34.370560] D [MSGID: 0] [afr-common.c:3188:afr_discover_do] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-replicate-0 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.370624] D [MSGID: 0] [dht-common.c:3013:dht_lookup_cbk] 0-gvol0-dht: fresh_lookup returned for / with op_ret -1 [Transport endpoint is not connected]
[2019-05-22 23:03:34.372619] D [MSGID: 0] [afr-common.c:3188:afr_discover_do] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-replicate-0 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.372693] D [MSGID: 0] [dht-common.c:1303:dht_lookup_dir_cbk] 0-gvol0-dht: lookup of / on gvol0-replicate-0 returned error [Transport endpoint is not connected]
[2019-05-22 23:03:34.372737] D [MSGID: 0] [dht-common.c:1473:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-dht returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.372783] D [MSGID: 0] [write-behind.c:2433:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-write-behind returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.372826] D [MSGID: 0] [io-cache.c:263:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-io-cache returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.372870] D [MSGID: 0] [quick-read.c:627:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-quick-read returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.372908] D [MSGID: 0] [md-cache.c:1213:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-md-cache returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.372954] D [MSGID: 0] [io-stats.c:2214:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.373070] E [fuse-bridge.c:4336:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
[2019-05-22 23:03:34.373985] D [fuse-helpers.c:649:fuse_ignore_xattr_set] 0-glusterfs-fuse: allowing setxattr: key [trusted.add-brick],  client pid [-6]
[2019-05-22 23:03:34.374251] D [MSGID: 0] [dht-common.c:3454:dht_do_fresh_lookup] 0-gvol0-dht: Calling fresh lookup for / on gvol0-replicate-0
[2019-05-22 23:03:34.376220] D [MSGID: 0] [afr-common.c:3188:afr_discover_do] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-replicate-0 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.376281] D [MSGID: 0] [dht-common.c:3013:dht_lookup_cbk] 0-gvol0-dht: fresh_lookup returned for / with op_ret -1 [Transport endpoint is not connected]
[2019-05-22 23:03:34.378241] D [MSGID: 0] [afr-common.c:3188:afr_discover_do] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-replicate-0 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378296] D [MSGID: 0] [dht-common.c:1303:dht_lookup_dir_cbk] 0-gvol0-dht: lookup of / on gvol0-replicate-0 returned error [Transport endpoint is not connected]
[2019-05-22 23:03:34.378330] D [MSGID: 0] [dht-common.c:1473:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-dht returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378360] D [MSGID: 0] [write-behind.c:2433:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-write-behind returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378389] D [MSGID: 0] [io-cache.c:263:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-io-cache returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378421] D [MSGID: 0] [quick-read.c:627:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-quick-read returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378448] D [MSGID: 0] [md-cache.c:1213:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-md-cache returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378474] D [MSGID: 0] [io-stats.c:2214:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378526] W [fuse-resolve.c:127:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)
[2019-05-22 23:03:34.378565] W [fuse-bridge.c:3294:fuse_setxattr_resume] 0-glusterfs-fuse: 2: SETXATTR 00000000-0000-0000-0000-000000000001/1 (trusted.add-brick) resolution failed
[2019-05-22 23:03:34.394682] D [fuse-bridge.c:5038:fuse_thread_proc] 0-glusterfs-fuse: terminating upon getting ENODEV when reading /dev/fuse
[2019-05-22 23:03:34.394747] I [fuse-bridge.c:5144:fuse_thread_proc] 0-fuse: initating unmount of /tmp/mntBBQr8A
[2019-05-22 23:03:34.394830] D [logging.c:1805:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 5 extra log messages
[2019-05-22 23:03:34.394900] D [logging.c:1808:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 5 extra log messages
[2019-05-22 23:03:34.395005] W [glusterfsd.c:1500:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f54bec80dd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55be99fe4e75] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55be99fe4ceb] ) 0-: received signum (15), shutting down
[2019-05-22 23:03:34.395042] D [mgmt-pmap.c:86:rpc_clnt_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given
[2019-05-22 23:03:34.395065] I [fuse-bridge.c:5914:fini] 0-fuse: Unmounting '/tmp/mntBBQr8A'.
[2019-05-22 23:03:34.395085] I [fuse-bridge.c:5919:fini] 0-fuse: Closing fuse connection to '/tmp/mntBBQr8A'.
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux