Re: add-brick: failed: Commit failed

David Cunningham <dcunningham@xxxxxxxxxxxxx> · Thu, 23 May 2019 10:24:28 +1200

Hi Ravi,

Please see the log attached. The output of "gluster volume status" is as follows. Should there be something listening on gfs3? I'm not sure whether it having TCP Port and Pid as N/A is a symptom or cause. Thank you.

# gluster volume status
Status of volume: gvol0
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick gfs1:/nodirectwritedata/gluster/gvol0 49152     0          Y       7706 
Brick gfs2:/nodirectwritedata/gluster/gvol0 49152     0          Y       7624 
Brick gfs3:/nodirectwritedata/gluster/gvol0 N/A       N/A        N       N/A  
Self-heal Daemon on localhost               N/A       N/A        Y       19853
Self-heal Daemon on gfs1                    N/A       N/A        Y       28600
Self-heal Daemon on gfs2                    N/A       N/A        Y       17614

Task Status of Volume gvol0
------------------------------------------------------------------------------
There are no active volume tasks

On Wed, 22 May 2019 at 18:06, Ravishankar N <ravishankar@xxxxxxxxxx> wrote:

    If you are trying this again, please 'gluster volume set $volname
      client-log-level DEBUG`before attempting the add-brick and attach
      the gvol0-add-brick-mount.log here. After that, you can change the
      client-log-level back to INFO.
    -Ravi

    On 22/05/19 11:32 AM, Ravishankar N
      wrote:

      On 22/05/19 11:23 AM, David
        Cunningham wrote:

          Hi Ravi,

          I'd already done exactly that before, where step 3 was a
            simple 'rm -rf /nodirectwritedata/gluster/gvol0'. Have you
            another suggestion on what the cleanup or reformat should
            be?

      `rm -rf /nodirectwritedata/gluster/gvol0` does look okay to me
      David. Basically, '/nodirectwritedata/gluster/gvol0' must be empty
      and must not have any extended attributes set on it. Why
      fuse_first_lookup() is failing is a bit of a mystery to me at this
      point. :-(

      Regards,

      Ravi

          Thank you.

          On Wed, 22 May 2019 at
            13:56, Ravishankar N <ravishankar@xxxxxxxxxx>
            wrote:

              Hmm, so the volume info seems to indicate that the
                add-brick was successful but the gfid xattr is missing
                on the new brick (as are the actual files, barring the
                .glusterfs folder, according to your previous mail).
              Do you want to try removing and adding it again?

              1. `gluster volume remove-brick gvol0 replica 2
                gfs3:/nodirectwritedata/gluster/gvol0 force` from gfs1

              2. Check that gluster volume info is now back to a 1x2
                volume on all nodes and `gluster peer status` is 
                connected on all nodes.

              3. Cleanup or reformat
                '/nodirectwritedata/gluster/gvol0' on gfs3.

              4. `gluster volume add-brick gvol0 replica 3 arbiter 1
                gfs3:/nodirectwritedata/gluster/gvol0` from gfs1.
              5. Check that the files are getting healed on to the
                new brick.

              Thanks,

              Ravi

              On
                22/05/19 6:50 AM, David Cunningham wrote:

                            Hi Ravi,

                            Certainly. On the existing two nodes:

                            gfs1 # getfattr -d -m. -e hex
                              /nodirectwritedata/gluster/gvol0

                              getfattr: Removing leading '/' from
                              absolute path names

                              # file: nodirectwritedata/gluster/gvol0

trusted.afr.dirty=0x000000000000000000000000

trusted.afr.gvol0-client-2=0x000000000000000000000000

trusted.gfid=0x00000000000000000000000000000001

trusted.glusterfs.dht=0x000000010000000000000000ffffffff

trusted.glusterfs.volume-id=0xfb5af69e1c3e41648b23c1d7bec9b1b6

                            gfs2 # getfattr -d -m. -e hex
                              /nodirectwritedata/gluster/gvol0

                              getfattr: Removing leading '/' from
                              absolute path names

                              # file: nodirectwritedata/gluster/gvol0

trusted.afr.dirty=0x000000000000000000000000

trusted.afr.gvol0-client-0=0x000000000000000000000000

trusted.afr.gvol0-client-2=0x000000000000000000000000

trusted.gfid=0x00000000000000000000000000000001

trusted.glusterfs.dht=0x000000010000000000000000ffffffff

trusted.glusterfs.volume-id=0xfb5af69e1c3e41648b23c1d7bec9b1b6

                            On the new node:

                            gfs3 # getfattr -d -m. -e hex
                              /nodirectwritedata/gluster/gvol0

                              getfattr: Removing leading '/' from
                              absolute path names

                              # file: nodirectwritedata/gluster/gvol0

trusted.afr.dirty=0x000000000000000000000001

trusted.glusterfs.volume-id=0xfb5af69e1c3e41648b23c1d7bec9b1b6

                            Output of "gluster volume info" is the
                              same on all 3 nodes and is:

                            # gluster volume info

                              Volume Name: gvol0

                              Type: Replicate

                              Volume ID:
                              fb5af69e-1c3e-4164-8b23-c1d7bec9b1b6

                              Status: Started

                              Snapshot Count: 0

                              Number of Bricks: 1 x (2 + 1) = 3

                              Transport-type: tcp

                              Bricks:

                              Brick1:
                              gfs1:/nodirectwritedata/gluster/gvol0

                              Brick2:
                              gfs2:/nodirectwritedata/gluster/gvol0

                              Brick3:
                              gfs3:/nodirectwritedata/gluster/gvol0
                              (arbiter)

                              Options Reconfigured:

                              performance.client-io-threads: off

                              nfs.disable: on

                              transport.address-family: inet

                  On Wed, 22 May 2019
                    at 12:43, Ravishankar N <ravishankar@xxxxxxxxxx>
                    wrote:

                     Hi David,

                      Could you provide the `getfattr -d -m. -e hex
                      /nodirectwritedata/gluster/gvol0` output of all
                      bricks and the output of `gluster volume info`?

                      Thanks,

                      Ravi

                      On
                        22/05/19 4:57 AM, David Cunningham wrote:

                                    Hi Sanju,

                                    Here's what glusterd.log says
                                      on the new arbiter server when
                                      trying to add the node:

                                    [2019-05-22 00:15:05.963059] I
                                      [run.c:242:runner_log]
                                      (-->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0x3b2cd)
                                      [0x7fe4ca9102cd]
                                      -->/usr/lib64/glusterfs/5.6/xlator/mgmt/glusterd.so(+0xe6b85)
                                      [0x7fe4ca9bbb85]
                                      -->/lib64/libglusterfs.so.0(runner_log+0x115)
                                      [0x7fe4d5ecc955] ) 0-management:
                                      Ran script:
/var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh
                                      --volname=gvol0 --version=1
                                      --volume-op=add-brick
                                      --gd-workdir=/var/lib/glusterd

                                      [2019-05-22 00:15:05.963177] I
                                      [MSGID: 106578]
                                      [glusterd-brick-ops.c:1355:glusterd_op_perform_add_bricks]
                                      0-management: replica-count is set
                                      3

                                      [2019-05-22 00:15:05.963228] I
                                      [MSGID: 106578]
                                      [glusterd-brick-ops.c:1360:glusterd_op_perform_add_bricks]
                                      0-management: arbiter-count is set
                                      1

                                      [2019-05-22 00:15:05.963257] I
                                      [MSGID: 106578]
                                      [glusterd-brick-ops.c:1364:glusterd_op_perform_add_bricks]
                                      0-management: type is set 0, need
                                      to change it

                                      [2019-05-22 00:15:17.015268] E
                                      [MSGID: 106053]
                                      [glusterd-utils.c:13942:glusterd_handle_replicate_brick_ops]
                                      0-management: Failed to set
                                      extended attribute
                                      trusted.add-brick : Transport
                                      endpoint is not connected
                                      [Transport endpoint is not
                                      connected]

                                      [2019-05-22 00:15:17.036479] E
                                      [MSGID: 106073]
                                      [glusterd-brick-ops.c:2595:glusterd_op_add_brick]
                                      0-glusterd: Unable to add bricks

                                      [2019-05-22 00:15:17.036595] E
                                      [MSGID: 106122]
                                      [glusterd-mgmt.c:299:gd_mgmt_v3_commit_fn]
                                      0-management: Add-brick commit
                                      failed.

                                      [2019-05-22 00:15:17.036710] E
                                      [MSGID: 106122]
                                      [glusterd-mgmt-handler.c:594:glusterd_handle_commit_fn]
                                      0-management: commit failed on
                                      operation Add brick

                                    As before
                                      gvol0-add-brick-mount.log said:

                                    [2019-05-22 00:15:17.005695] I
                                      [fuse-bridge.c:4267:fuse_init]
                                      0-glusterfs-fuse: FUSE inited with
                                      protocol versions: glusterfs 7.24
                                      kernel 7.22

                                      [2019-05-22 00:15:17.005749] I
                                      [fuse-bridge.c:4878:fuse_graph_sync]
                                      0-fuse: switched to graph 0

                                      [2019-05-22 00:15:17.010101] E
                                      [fuse-bridge.c:4336:fuse_first_lookup]
                                      0-fuse: first lookup on root
                                      failed (Transport endpoint is not
                                      connected)

                                      [2019-05-22 00:15:17.014217] W
                                      [fuse-bridge.c:897:fuse_attr_cbk]
                                      0-glusterfs-fuse: 2: LOOKUP() /
                                      => -1 (Transport endpoint is
                                      not connected)

                                      [2019-05-22 00:15:17.015097] W
                                      [fuse-resolve.c:127:fuse_resolve_gfid_cbk]
                                      0-fuse:
                                      00000000-0000-0000-0000-000000000001:
                                      failed to resolve (Transport
                                      endpoint is not connected)

                                      [2019-05-22 00:15:17.015158] W
                                      [fuse-bridge.c:3294:fuse_setxattr_resume]
                                      0-glusterfs-fuse: 3: SETXATTR
                                      00000000-0000-0000-0000-000000000001/1
                                      (trusted.add-brick) resolution
                                      failed

                                      [2019-05-22 00:15:17.035636] I
                                      [fuse-bridge.c:5144:fuse_thread_proc]
                                      0-fuse: initating unmount of
                                      /tmp/mntYGNbj9

                                      [2019-05-22 00:15:17.035854] W
                                      [glusterfsd.c:1500:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7f7745ccedd5]
                                      -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5)
                                      [0x55c81b63de75]
                                      -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b)
                                      [0x55c81b63dceb] ) 0-: received
                                      signum (15), shutting down

                                      [2019-05-22 00:15:17.035942] I
                                      [fuse-bridge.c:5914:fini] 0-fuse:
                                      Unmounting '/tmp/mntYGNbj9'.

                                      [2019-05-22 00:15:17.035966] I
                                      [fuse-bridge.c:5919:fini] 0-fuse:
                                      Closing fuse connection to
                                      '/tmp/mntYGNbj9'.

                                    Here are the processes running
                                      on the new arbiter server:
                                    # ps -ef | grep gluster

                                      root      3466     1  0 20:13
                                      ?        00:00:00
                                      /usr/sbin/glusterfs -s localhost
                                      --volfile-id gluster/glustershd -p
/var/run/gluster/glustershd/glustershd.pid -l
                                      /var/log/glusterfs/glustershd.log
                                      -S
                                      /var/run/gluster/24c12b09f93eec8e.socket
                                      --xlator-option
                                      *replicate*.node-uuid=2069cfb3-c798-47e3-8cf8-3c584cf7c412
                                      --process-name glustershd

                                      root      6832     1  0 May16
                                      ?        00:02:10
                                      /usr/sbin/glusterd -p
                                      /var/run/glusterd.pid --log-level
                                      INFO

                                      root     17841     1  0 May16
                                      ?        00:00:58
                                      /usr/sbin/glusterfs --process-name
                                      fuse --volfile-server=gfs1
                                      --volfile-id=/gvol0 /mnt/glusterfs

                                    Here are the files created on
                                      the new arbiter server:
                                    # find
                                      /nodirectwritedata/gluster/gvol0 |
                                      xargs ls -ald

                                      drwxr-xr-x 3 root root 4096 May 21
                                      20:15
                                      /nodirectwritedata/gluster/gvol0

                                      drw------- 2 root root 4096 May 21
                                      20:15
                                      /nodirectwritedata/gluster/gvol0/.glusterfs

                                    Thank you for your help!

                          On Tue, 21
                            May 2019 at 00:10, Sanju Rakonde <srakonde@xxxxxxxxxx>
                            wrote:

                            David,

                              can you please attach glusterd.logs?
                                As the error message says, Commit failed
                                on the arbitar node, we might be able to
                                find some issue on that node.

                              On Mon,
                                May 20, 2019 at 10:10 AM Nithya
                                Balachandran <nbalacha@xxxxxxxxxx>
                                wrote:

                                    On
                                      Fri, 17 May 2019 at 06:01, David
                                      Cunningham <dcunningham@xxxxxxxxxxxxx>
                                      wrote:

                                                      Hello,

                                                      We're adding
                                                        an arbiter node
                                                        to an existing
                                                        volume and
                                                        having an issue.
                                                        Can anyone help?
                                                        The root cause
                                                        error appears to
                                                        be
"00000000-0000-0000-0000-000000000001: failed to resolve (Transport
                                                        endpoint is not
                                                        connected)", as
                                                        below.

                                                      We are
                                                        running
                                                        glusterfs 5.6.1.
                                                        Thanks in
                                                        advance for any
                                                        assistance!

                                                      On existing
                                                        node gfs1,
                                                        trying to add
                                                        new arbiter node
                                                        gfs3:

                                                      # gluster
                                                        volume add-brick
                                                        gvol0 replica 3
                                                        arbiter 1
                                                        gfs3:/nodirectwritedata/gluster/gvol0

                                                        volume
                                                        add-brick:
                                                        failed: Commit
                                                        failed on gfs3.
                                                        Please check log
                                                        file for
                                                        details.

                                    This looks like a glusterd
                                      issue. Please check the glusterd
                                      logs for more info.
                                    Adding the glusterd dev to this
                                      thread. Sanju, can you take a
                                      look?

                                    Regards,
                                    Nithya

                                                      On new node
                                                        gfs3 in
                                                        gvol0-add-brick-mount.log:

                                                      [2019-05-17
                                                        01:20:22.689721]
                                                        I
[fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse: FUSE inited with
                                                        protocol
                                                        versions:
                                                        glusterfs 7.24
                                                        kernel 7.22

                                                        [2019-05-17
                                                        01:20:22.689778]
                                                        I
[fuse-bridge.c:4878:fuse_graph_sync] 0-fuse: switched to graph 0

                                                        [2019-05-17
                                                        01:20:22.694897]
                                                        E
[fuse-bridge.c:4336:fuse_first_lookup] 0-fuse: first lookup on root
                                                        failed
                                                        (Transport
                                                        endpoint is not
                                                        connected)

                                                        [2019-05-17
                                                        01:20:22.699770]
                                                        W
[fuse-resolve.c:127:fuse_resolve_gfid_cbk] 0-fuse:
                                                        00000000-0000-0000-0000-000000000001:
                                                        failed to
                                                        resolve
                                                        (Transport
                                                        endpoint is not
                                                        connected)

                                                        [2019-05-17
                                                        01:20:22.699834]
                                                        W
[fuse-bridge.c:3294:fuse_setxattr_resume] 0-glusterfs-fuse: 2: SETXATTR
00000000-0000-0000-0000-000000000001/1 (trusted.add-brick) resolution
                                                        failed

                                                        [2019-05-17
                                                        01:20:22.715656]
                                                        I
[fuse-bridge.c:5144:fuse_thread_proc] 0-fuse: initating unmount of
                                                        /tmp/mntQAtu3f

                                                        [2019-05-17
                                                        01:20:22.715865]
                                                        W
[glusterfsd.c:1500:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7fb223bf6dd5]
                                                        -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5)
                                                        [0x560886581e75]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x560886581ceb] ) 0-:
                                                        received signum
                                                        (15), shutting
                                                        down

                                                        [2019-05-17
                                                        01:20:22.715926]
                                                        I
[fuse-bridge.c:5914:fini] 0-fuse: Unmounting '/tmp/mntQAtu3f'.

                                                        [2019-05-17
                                                        01:20:22.715953]
                                                        I
[fuse-bridge.c:5919:fini] 0-fuse: Closing fuse connection to
                                                        '/tmp/mntQAtu3f'.

                                                      Processes
                                                        running on new
                                                        node gfs3:

                                                      # ps -ef |
                                                        grep gluster

                                                        root     
                                                        6832     1  0
                                                        20:17 ?       
                                                        00:00:00
                                                        /usr/sbin/glusterd
                                                        -p
                                                        /var/run/glusterd.pid
                                                        --log-level INFO

                                                        root    
                                                        15799     1  0
                                                        20:17 ?       
                                                        00:00:00
/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p
/var/run/gluster/glustershd/glustershd.pid -l
                                                        /var/log/glusterfs/glustershd.log
                                                        -S
                                                        /var/run/gluster/24c12b09f93eec8e.socket
                                                        --xlator-option
*replicate*.node-uuid=2069cfb3-c798-47e3-8cf8-3c584cf7c412
                                                        --process-name
                                                        glustershd

                                                        root     16856
                                                        16735  0 21:21
                                                        pts/0   
                                                        00:00:00 grep
                                                        --color=auto
                                                        gluster

                                                      -- 

                                                          David
                                                          Cunningham,
                                                          Voisonics
                                                          Limited

                                                          http://voisonics.com/

                                                          USA: +1 213
                                                          221 1092

                                                          New Zealand:
                                                          +64 (0)28 2558
                                                          3782

_______________________________________________

                                      Gluster-users mailing list

                                      Gluster-users@xxxxxxxxxxx

                                      https://lists.gluster.org/mailman/listinfo/gluster-users

                            -- 

                                Thanks,

                                Sanju

                        -- 

                                            David Cunningham,
                                              Voisonics Limited

                                              http://voisonics.com/

                                              USA: +1 213 221 1092

                                              New Zealand: +64 (0)28
                                              2558 3782

                        _______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users

                -- 

                                    David Cunningham, Voisonics
                                      Limited

                                      http://voisonics.com/

                                      USA: +1 213 221 1092

                                      New Zealand: +64 (0)28 2558 3782

        -- 

                            David Cunningham, Voisonics Limited

                              http://voisonics.com/

                              USA: +1 213 221 1092

                              New Zealand: +64 (0)28 2558 3782

-- 
David Cunningham, Voisonics Limited
http://voisonics.com/
USA: +1 213 221 1092
New Zealand: +64 (0)28 2558 3782
[2019-05-22 23:03:23.344284] I [MSGID: 100030] [glusterfsd.c:2725:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 5.6 (args: /usr/sbin/glusterfs --volfile /tmp/gvol0.tcp-fuse.vol --client-pid -6 -l /var/log/glusterfs/gvol0-add-brick-mount.log /tmp/mntBBQr8A)
[2019-05-22 23:03:23.351566] D [MSGID: 0] [quick-read.c:1183:check_cache_size_ok] 0-gvol0-quick-read: Max cache size is 67334021120
[2019-05-22 23:03:23.351743] D [MSGID: 0] [io-cache.c:1595:check_cache_size_ok] 0-gvol0-io-cache: Max cache size is 67334021120
[2019-05-22 23:03:23.351809] D [MSGID: 0] [options.c:1227:xlator_option_init_size_uint64] 0-gvol0-readdir-ahead: option rda-request-size using set value 131072
[2019-05-22 23:03:23.351844] D [MSGID: 0] [options.c:1227:xlator_option_init_size_uint64] 0-gvol0-readdir-ahead: option rda-cache-limit using set value 10MB
[2019-05-22 23:03:23.351870] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-gvol0-readdir-ahead: option parallel-readdir using set value off
[2019-05-22 23:03:23.352098] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-gvol0-dht: option lock-migration using set value off
[2019-05-22 23:03:23.352143] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-gvol0-dht: option force-migration using set value off
[2019-05-22 23:03:23.352251] D [MSGID: 0] [dht-shared.c:356:dht_init_regex] 0-gvol0-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
[2019-05-22 23:03:23.352376] D [MSGID: 0] [options.c:1224:xlator_option_init_uint32] 0-gvol0-replicate-0: option arbiter-count using set value 1
[2019-05-22 23:03:23.352810] D [MSGID: 0] [options.c:1221:xlator_option_init_str] 0-gvol0-replicate-0: option afr-pending-xattr using set value gvol0-client-0,gvol0-client-1,gvol0-client-2
[2019-05-22 23:03:23.354194] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-gvol0-client-2: option ping-timeout using set value 42
[2019-05-22 23:03:23.354231] D [MSGID: 0] [options.c:1232:xlator_option_init_path] 0-gvol0-client-2: option remote-subvolume using set value /nodirectwritedata/gluster/gvol0
[2019-05-22 23:03:23.354263] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-gvol0-client-2: option send-gids using set value true
[2019-05-22 23:03:23.354325] D [rpc-clnt.c:1002:rpc_clnt_connection_init] 0-gvol0-client-2: defaulting frame-timeout to 30mins
[2019-05-22 23:03:23.354352] D [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-gvol0-client-2: setting ping-timeout to 42
[2019-05-22 23:03:23.354387] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.6/rpc-transport/socket.so
[2019-05-22 23:03:23.361071] D [socket.c:4466:socket_init] 0-gvol0-client-2: Configued transport.tcp-user-timeout=0
[2019-05-22 23:03:23.361108] D [socket.c:4484:socket_init] 0-gvol0-client-2: Reconfigued transport.keepalivecnt=9
[2019-05-22 23:03:23.361129] D [socket.c:4169:ssl_setup_connection_params] 0-gvol0-client-2: SSL support on the I/O path is NOT enabled
[2019-05-22 23:03:23.361144] D [socket.c:4172:ssl_setup_connection_params] 0-gvol0-client-2: SSL support for glusterd is NOT enabled
[2019-05-22 23:03:23.361178] D [rpc-clnt.c:1579:rpcclnt_cbk_program_register] 0-gvol0-client-2: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
[2019-05-22 23:03:23.361200] D [MSGID: 0] [client.c:2547:client_init_rpc] 0-gvol0-client-2: client init successful
[2019-05-22 23:03:23.361271] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-gvol0-client-1: option ping-timeout using set value 42
[2019-05-22 23:03:23.361303] D [MSGID: 0] [options.c:1232:xlator_option_init_path] 0-gvol0-client-1: option remote-subvolume using set value /nodirectwritedata/gluster/gvol0
[2019-05-22 23:03:23.361336] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-gvol0-client-1: option send-gids using set value true
[2019-05-22 23:03:23.361357] D [rpc-clnt.c:1002:rpc_clnt_connection_init] 0-gvol0-client-1: defaulting frame-timeout to 30mins
[2019-05-22 23:03:23.361371] D [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-gvol0-client-1: setting ping-timeout to 42
[2019-05-22 23:03:23.361390] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.6/rpc-transport/socket.so
[2019-05-22 23:03:23.361564] D [socket.c:4466:socket_init] 0-gvol0-client-1: Configued transport.tcp-user-timeout=0
[2019-05-22 23:03:23.361591] D [socket.c:4484:socket_init] 0-gvol0-client-1: Reconfigued transport.keepalivecnt=9
[2019-05-22 23:03:23.361605] D [socket.c:4169:ssl_setup_connection_params] 0-gvol0-client-1: SSL support on the I/O path is NOT enabled
[2019-05-22 23:03:23.361617] D [socket.c:4172:ssl_setup_connection_params] 0-gvol0-client-1: SSL support for glusterd is NOT enabled
[2019-05-22 23:03:23.361630] D [rpc-clnt.c:1579:rpcclnt_cbk_program_register] 0-gvol0-client-1: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
[2019-05-22 23:03:23.361649] D [MSGID: 0] [client.c:2547:client_init_rpc] 0-gvol0-client-1: client init successful
[2019-05-22 23:03:23.361699] D [MSGID: 0] [options.c:1225:xlator_option_init_int32] 0-gvol0-client-0: option ping-timeout using set value 42
[2019-05-22 23:03:23.361734] D [MSGID: 0] [options.c:1232:xlator_option_init_path] 0-gvol0-client-0: option remote-subvolume using set value /nodirectwritedata/gluster/gvol0
[2019-05-22 23:03:23.361762] D [MSGID: 0] [options.c:1230:xlator_option_init_bool] 0-gvol0-client-0: option send-gids using set value true
[2019-05-22 23:03:23.361784] D [rpc-clnt.c:1002:rpc_clnt_connection_init] 0-gvol0-client-0: defaulting frame-timeout to 30mins
[2019-05-22 23:03:23.361797] D [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-gvol0-client-0: setting ping-timeout to 42
[2019-05-22 23:03:23.361829] D [rpc-transport.c:269:rpc_transport_load] 0-rpc-transport: attempt to load file /usr/lib64/glusterfs/5.6/rpc-transport/socket.so
[2019-05-22 23:03:23.361984] D [socket.c:4466:socket_init] 0-gvol0-client-0: Configued transport.tcp-user-timeout=0
[2019-05-22 23:03:23.362004] D [socket.c:4484:socket_init] 0-gvol0-client-0: Reconfigued transport.keepalivecnt=9
[2019-05-22 23:03:23.362018] D [socket.c:4169:ssl_setup_connection_params] 0-gvol0-client-0: SSL support on the I/O path is NOT enabled
[2019-05-22 23:03:23.362031] D [socket.c:4172:ssl_setup_connection_params] 0-gvol0-client-0: SSL support for glusterd is NOT enabled
[2019-05-22 23:03:23.362046] D [rpc-clnt.c:1579:rpcclnt_cbk_program_register] 0-gvol0-client-0: New program registered: GlusterFS Callback, Num: 52743234, Ver: 1
[2019-05-22 23:03:23.362060] D [MSGID: 0] [client.c:2547:client_init_rpc] 0-gvol0-client-0: client init successful
[2019-05-22 23:03:23.362168] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-2: option 'transport.address-family' is not recognized
[2019-05-22 23:03:23.362194] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-2: option 'transport.tcp-user-timeout' is not recognized
[2019-05-22 23:03:23.362218] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-2: option 'transport.socket.keepalive-time' is not recognized
[2019-05-22 23:03:23.362239] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-2: option 'transport.socket.keepalive-interval' is not recognized
[2019-05-22 23:03:23.362260] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-2: option 'transport.socket.keepalive-count' is not recognized
[2019-05-22 23:03:23.362293] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-1: option 'transport.address-family' is not recognized
[2019-05-22 23:03:23.362317] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-1: option 'transport.tcp-user-timeout' is not recognized
[2019-05-22 23:03:23.362338] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-1: option 'transport.socket.keepalive-time' is not recognized
[2019-05-22 23:03:23.362358] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-1: option 'transport.socket.keepalive-interval' is not recognized
[2019-05-22 23:03:23.362380] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-1: option 'transport.socket.keepalive-count' is not recognized
[2019-05-22 23:03:23.362427] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-0: option 'transport.address-family' is not recognized
[2019-05-22 23:03:23.362447] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-0: option 'transport.tcp-user-timeout' is not recognized
[2019-05-22 23:03:23.362468] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-0: option 'transport.socket.keepalive-time' is not recognized
[2019-05-22 23:03:23.362502] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-0: option 'transport.socket.keepalive-interval' is not recognized
[2019-05-22 23:03:23.362530] D [MSGID: 101174] [graph.c:397:_log_if_unknown_option] 0-gvol0-client-0: option 'transport.socket.keepalive-count' is not recognized
[2019-05-22 23:03:23.362561] D [fuse-bridge.c:5332:notify] 0-fuse: got event 12 on graph 0
[2019-05-22 23:03:23.362615] D [MSGID: 0] [afr-common.c:5012:__afr_launch_notify_timer] 0-gvol0-replicate-0: Initiating child-down timer
[2019-05-22 23:03:23.362652] I [MSGID: 114020] [client.c:2358:notify] 0-gvol0-client-0: parent translators are ready, attempting connect on transport
[2019-05-22 23:03:23.367816] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-69.42.167.137 (port-24007) for hostname: gfs1 and port: 24007
[2019-05-22 23:03:23.367850] D [socket.c:3223:socket_fix_ssl_opts] 0-gvol0-client-0: disabling SSL for portmapper connection
[2019-05-22 23:03:23.368114] I [MSGID: 114020] [client.c:2358:notify] 0-gvol0-client-1: parent translators are ready, attempting connect on transport
[2019-05-22 23:03:23.372391] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-69.42.172.137 (port-24007) for hostname: gfs2 and port: 24007
[2019-05-22 23:03:23.372421] D [socket.c:3223:socket_fix_ssl_opts] 0-gvol0-client-1: disabling SSL for portmapper connection
[2019-05-22 23:03:23.372535] I [MSGID: 114020] [client.c:2358:notify] 0-gvol0-client-2: parent translators are ready, attempting connect on transport
[2019-05-22 23:03:23.376795] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-192.157.88.220 (port-24007) for hostname: gfs3 and port: 24007
[2019-05-22 23:03:23.376825] D [socket.c:3223:socket_fix_ssl_opts] 0-gvol0-client-2: disabling SSL for portmapper connection
[2019-05-22 23:03:23.378012] I [MSGID: 101190] [event-epoll.c:621:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2019-05-22 23:03:23.378090] I [MSGID: 101190] [event-epoll.c:621:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2019-05-22 23:03:23.378143] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-0: got RPC_CLNT_CONNECT
[2019-05-22 23:03:23.378154] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-1: got RPC_CLNT_CONNECT
[2019-05-22 23:03:23.378862] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 69.42.172.137:24007: ping timer event already removed
[2019-05-22 23:03:23.378881] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 69.42.167.137:24007: ping timer event already removed
[2019-05-22 23:03:23.378971] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_CONNECT
[2019-05-22 23:03:23.379191] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 192.157.88.220:24007: ping timer event already removed
[2019-05-22 23:03:23.379300] D [MSGID: 0] [client-handshake.c:1392:server_has_portmap] 0-gvol0-client-2: detected portmapper on server
[2019-05-22 23:03:23.379306] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-2: Ping latency is 0ms
[2019-05-22 23:03:23.379445] E [MSGID: 114058] [client-handshake.c:1449:client_query_portmap_cbk] 0-gvol0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2019-05-22 23:03:23.379512] D [socket.c:2929:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:13) (non-SSL)
[2019-05-22 23:03:23.379544] D [MSGID: 0] [client.c:2275:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_DISCONNECT
[2019-05-22 23:03:23.379562] I [MSGID: 114018] [client.c:2288:client_rpc_notify] 0-gvol0-client-2: disconnected from gvol0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2019-05-22 23:03:23.379586] E [MSGID: 108006] [afr-common.c:5314:__afr_handle_child_down_event] 0-gvol0-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.
[2019-05-22 23:03:23.379594] D [MSGID: 0] [client-handshake.c:1392:server_has_portmap] 0-gvol0-client-0: detected portmapper on server
[2019-05-22 23:03:23.380198] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-0: Ping latency is 1ms
[2019-05-22 23:03:23.380367] D [MSGID: 0] [client-handshake.c:1392:server_has_portmap] 0-gvol0-client-1: detected portmapper on server
[2019-05-22 23:03:23.380892] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-gvol0-client-0: changing port to 49152 (from 0)
[2019-05-22 23:03:23.380927] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-1: Ping latency is 2ms
[2019-05-22 23:03:23.380951] D [socket.c:2929:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:11) (non-SSL)
[2019-05-22 23:03:23.380974] D [MSGID: 0] [client.c:2275:client_rpc_notify] 0-gvol0-client-0: got RPC_CLNT_DISCONNECT
[2019-05-22 23:03:23.380992] D [MSGID: 0] [client.c:2316:client_rpc_notify] 0-gvol0-client-0: disconnected (skipped notify)
[2019-05-22 23:03:23.382318] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-gvol0-client-1: changing port to 49152 (from 0)
[2019-05-22 23:03:23.382377] D [socket.c:2929:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:12) (non-SSL)
[2019-05-22 23:03:23.382398] D [MSGID: 0] [client.c:2275:client_rpc_notify] 0-gvol0-client-1: got RPC_CLNT_DISCONNECT
[2019-05-22 23:03:23.382417] D [MSGID: 0] [client.c:2316:client_rpc_notify] 0-gvol0-client-1: disconnected (skipped notify)
[2019-05-22 23:03:23.384934] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-69.42.167.137 (port-24007) for hostname: gfs1 and port: 24007
[2019-05-22 23:03:23.386356] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-0: got RPC_CLNT_CONNECT
[2019-05-22 23:03:23.386603] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 69.42.167.137:49152: ping timer event already removed
[2019-05-22 23:03:23.387777] W [dict.c:1002:str_to_data] (-->/usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x403f7) [0x7f54b478d3f7] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f54bfe15b76] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f54bfe12491] ) 0-dict: value is NULL [Invalid argument]
[2019-05-22 23:03:23.387811] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-gvol0-client-0: failed to set process-name in handshake msg
[2019-05-22 23:03:23.387909] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-0: Ping latency is 1ms
[2019-05-22 23:03:23.388792] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-69.42.172.137 (port-24007) for hostname: gfs2 and port: 24007
[2019-05-22 23:03:23.389716] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-gvol0-client-0: Connected to gvol0-client-0, attached to remote volume '/nodirectwritedata/gluster/gvol0'.
[2019-05-22 23:03:23.389754] D [MSGID: 0] [client-handshake.c:945:client_post_handshake] 0-gvol0-client-0: No fds to open - notifying all parents child up
[2019-05-22 23:03:23.389780] D [MSGID: 0] [afr-common.c:5157:afr_get_halo_latency] 0-gvol0-replicate-0: Using halo latency 5
[2019-05-22 23:03:23.389811] I [MSGID: 108005] [afr-common.c:5237:__afr_handle_child_up_event] 0-gvol0-replicate-0: Subvolume 'gvol0-client-0' came back up; going online.
[2019-05-22 23:03:27.352904] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-192.157.88.220 (port-24007) for hostname: gfs3 and port: 24007
[2019-05-22 23:03:27.352959] D [socket.c:3223:socket_fix_ssl_opts] 0-gvol0-client-2: disabling SSL for portmapper connection
[2019-05-22 23:03:27.353228] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_CONNECT
[2019-05-22 23:03:27.353608] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 192.157.88.220:24007: ping timer event already removed
[2019-05-22 23:03:27.353741] D [MSGID: 0] [client-handshake.c:1392:server_has_portmap] 0-gvol0-client-2: detected portmapper on server
[2019-05-22 23:03:27.353853] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-2: Ping latency is 0ms
[2019-05-22 23:03:27.353922] D [MSGID: 0] [client-handshake.c:1455:client_query_portmap_cbk] 0-gvol0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2019-05-22 23:03:27.353997] D [socket.c:2929:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:11) (non-SSL)
[2019-05-22 23:03:27.354042] D [MSGID: 0] [client.c:2275:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_DISCONNECT
[2019-05-22 23:03:27.354073] D [MSGID: 0] [client.c:2296:client_rpc_notify] 0-gvol0-client-2: disconnected from gvol0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2019-05-22 23:03:30.359183] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-192.157.88.220 (port-24007) for hostname: gfs3 and port: 24007
[2019-05-22 23:03:30.359242] D [socket.c:3223:socket_fix_ssl_opts] 0-gvol0-client-2: disabling SSL for portmapper connection
[2019-05-22 23:03:30.359481] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_CONNECT
[2019-05-22 23:03:30.359846] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 192.157.88.220:24007: ping timer event already removed
[2019-05-22 23:03:30.359982] D [MSGID: 0] [client-handshake.c:1392:server_has_portmap] 0-gvol0-client-2: detected portmapper on server
[2019-05-22 23:03:30.360096] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-2: Ping latency is 0ms
[2019-05-22 23:03:30.360170] D [MSGID: 0] [client-handshake.c:1455:client_query_portmap_cbk] 0-gvol0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2019-05-22 23:03:30.360248] D [socket.c:2929:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:11) (non-SSL)
[2019-05-22 23:03:30.360298] D [MSGID: 0] [client.c:2275:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_DISCONNECT
[2019-05-22 23:03:30.360332] D [MSGID: 0] [client.c:2296:client_rpc_notify] 0-gvol0-client-2: disconnected from gvol0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2019-05-22 23:03:33.365443] D [MSGID: 0] [common-utils.c:536:gf_resolve_ip6] 0-resolver: returning ip-192.157.88.220 (port-24007) for hostname: gfs3 and port: 24007
[2019-05-22 23:03:33.365507] D [socket.c:3223:socket_fix_ssl_opts] 0-gvol0-client-2: disabling SSL for portmapper connection
[2019-05-22 23:03:33.365780] D [MSGID: 0] [client.c:2264:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_CONNECT
[2019-05-22 23:03:33.366149] D [rpc-clnt-ping.c:96:rpc_clnt_remove_ping_timer_locked] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f54bfe1efbb] (--> /lib64/libgfrpc.so.0(+0x125bb)[0x7f54bfbed5bb] (--> /lib64/libgfrpc.so.0(+0x12d81)[0x7f54bfbedd81] (--> /lib64/libgfrpc.so.0(rpc_clnt_submit+0x4bb)[0x7f54bfbea5fb] (--> /usr/lib64/glusterfs/5.6/xlator/protocol/client.so(+0x13f92)[0x7f54b4760f92] ))))) 0-: 192.157.88.220:24007: ping timer event already removed
[2019-05-22 23:03:33.366275] D [MSGID: 0] [client-handshake.c:1392:server_has_portmap] 0-gvol0-client-2: detected portmapper on server
[2019-05-22 23:03:33.366332] D [rpc-clnt-ping.c:204:rpc_clnt_ping_cbk] 0-gvol0-client-2: Ping latency is 0ms
[2019-05-22 23:03:33.366537] D [MSGID: 0] [client-handshake.c:1455:client_query_portmap_cbk] 0-gvol0-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2019-05-22 23:03:33.366625] D [socket.c:2929:socket_event_handler] 0-transport: EPOLLERR - disconnecting (sock:11) (non-SSL)
[2019-05-22 23:03:33.366658] D [MSGID: 0] [client.c:2275:client_rpc_notify] 0-gvol0-client-2: got RPC_CLNT_DISCONNECT
[2019-05-22 23:03:33.366687] D [MSGID: 0] [client.c:2296:client_rpc_notify] 0-gvol0-client-2: disconnected from gvol0-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2019-05-22 23:03:34.365978] D [fuse-bridge.c:5332:notify] 0-fuse: got event 5 on graph 0
[2019-05-22 23:03:34.367664] D [MSGID: 0] [dht-diskusage.c:94:dht_du_info_cbk] 0-gvol0-dht: subvolume 'gvol0-replicate-0': avail_percent is: 82.00 and avail_space is: 1554380546048 and avail_inodes is: 98.00
[2019-05-22 23:03:34.367800] D [fuse-bridge.c:4919:fuse_get_mount_status] 0-fuse: mount status is 0
[2019-05-22 23:03:34.368012] D [fuse-bridge.c:4206:fuse_init] 0-glusterfs-fuse: Detected support for FUSE_AUTO_INVAL_DATA. Enabling fopen_keep_cache automatically.
[2019-05-22 23:03:34.368073] I [fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2019-05-22 23:03:34.368102] I [fuse-bridge.c:4878:fuse_graph_sync] 0-fuse: switched to graph 0
[2019-05-22 23:03:34.368323] D [MSGID: 0] [dht-common.c:3454:dht_do_fresh_lookup] 0-gvol0-dht: Calling fresh lookup for / on gvol0-replicate-0
[2019-05-22 23:03:34.370560] D [MSGID: 0] [afr-common.c:3188:afr_discover_do] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-replicate-0 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.370624] D [MSGID: 0] [dht-common.c:3013:dht_lookup_cbk] 0-gvol0-dht: fresh_lookup returned for / with op_ret -1 [Transport endpoint is not connected]
[2019-05-22 23:03:34.372619] D [MSGID: 0] [afr-common.c:3188:afr_discover_do] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-replicate-0 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.372693] D [MSGID: 0] [dht-common.c:1303:dht_lookup_dir_cbk] 0-gvol0-dht: lookup of / on gvol0-replicate-0 returned error [Transport endpoint is not connected]
[2019-05-22 23:03:34.372737] D [MSGID: 0] [dht-common.c:1473:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-dht returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.372783] D [MSGID: 0] [write-behind.c:2433:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-write-behind returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.372826] D [MSGID: 0] [io-cache.c:263:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-io-cache returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.372870] D [MSGID: 0] [quick-read.c:627:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-quick-read returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.372908] D [MSGID: 0] [md-cache.c:1213:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-md-cache returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.372954] D [MSGID: 0] [io-stats.c:2214:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.373070] E [fuse-bridge.c:4336:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
[2019-05-22 23:03:34.373985] D [fuse-helpers.c:649:fuse_ignore_xattr_set] 0-glusterfs-fuse: allowing setxattr: key [trusted.add-brick],  client pid [-6]
[2019-05-22 23:03:34.374251] D [MSGID: 0] [dht-common.c:3454:dht_do_fresh_lookup] 0-gvol0-dht: Calling fresh lookup for / on gvol0-replicate-0
[2019-05-22 23:03:34.376220] D [MSGID: 0] [afr-common.c:3188:afr_discover_do] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-replicate-0 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.376281] D [MSGID: 0] [dht-common.c:3013:dht_lookup_cbk] 0-gvol0-dht: fresh_lookup returned for / with op_ret -1 [Transport endpoint is not connected]
[2019-05-22 23:03:34.378241] D [MSGID: 0] [afr-common.c:3188:afr_discover_do] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-replicate-0 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378296] D [MSGID: 0] [dht-common.c:1303:dht_lookup_dir_cbk] 0-gvol0-dht: lookup of / on gvol0-replicate-0 returned error [Transport endpoint is not connected]
[2019-05-22 23:03:34.378330] D [MSGID: 0] [dht-common.c:1473:dht_lookup_dir_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-dht returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378360] D [MSGID: 0] [write-behind.c:2433:wb_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-write-behind returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378389] D [MSGID: 0] [io-cache.c:263:ioc_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-io-cache returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378421] D [MSGID: 0] [quick-read.c:627:qr_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-quick-read returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378448] D [MSGID: 0] [md-cache.c:1213:mdc_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0-md-cache returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378474] D [MSGID: 0] [io-stats.c:2214:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f5498001048, gvol0 returned -1 error: Transport endpoint is not connected [Transport endpoint is not connected]
[2019-05-22 23:03:34.378526] W [fuse-resolve.c:127:fuse_resolve_gfid_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001: failed to resolve (Transport endpoint is not connected)
[2019-05-22 23:03:34.378565] W [fuse-bridge.c:3294:fuse_setxattr_resume] 0-glusterfs-fuse: 2: SETXATTR 00000000-0000-0000-0000-000000000001/1 (trusted.add-brick) resolution failed
[2019-05-22 23:03:34.394682] D [fuse-bridge.c:5038:fuse_thread_proc] 0-glusterfs-fuse: terminating upon getting ENODEV when reading /dev/fuse
[2019-05-22 23:03:34.394747] I [fuse-bridge.c:5144:fuse_thread_proc] 0-fuse: initating unmount of /tmp/mntBBQr8A
[2019-05-22 23:03:34.394830] D [logging.c:1805:gf_log_flush_extra_msgs] 0-logging-infra: Log buffer size reduced. About to flush 5 extra log messages
[2019-05-22 23:03:34.394900] D [logging.c:1808:gf_log_flush_extra_msgs] 0-logging-infra: Just flushed 5 extra log messages
[2019-05-22 23:03:34.395005] W [glusterfsd.c:1500:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f54bec80dd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55be99fe4e75] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55be99fe4ceb] ) 0-: received signum (15), shutting down
[2019-05-22 23:03:34.395042] D [mgmt-pmap.c:86:rpc_clnt_mgmt_pmap_signout] 0-fsd-mgmt: portmapper signout arguments not given
[2019-05-22 23:03:34.395065] I [fuse-bridge.c:5914:fini] 0-fuse: Unmounting '/tmp/mntBBQr8A'.
[2019-05-22 23:03:34.395085] I [fuse-bridge.c:5919:fini] 0-fuse: Closing fuse connection to '/tmp/mntBBQr8A'.
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users