Re: Problem with SSL/TLS encryption on Gluster 4.0 & 4.1

Amar Tumballi <atumball@xxxxxxxxxx> · Fri, 13 Jul 2018 15:57:56 +0530

Thanks David Spisla for pitching in.
Havriliuc, if you can file the bug as requested, we would like to check this out.

-Amar

On Fri, Jul 13, 2018 at 3:44 PM, David Spisla <spisla80@xxxxxxxxx> wrote:
Hello Andrei,

I did some research. But I didn't find a bug report for this issue. If you want you can create such a report here:
https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS

It is the recommended way in GlusterFS to report a bug, github issues are for new features

Regards
David

2018-07-13 11:44 GMT+02:00 Havriliuc Andrei <andrei@xxxxxxxxxxxxx>:

    Hello David,

    Yeah. I also disabled SSL/TLS for management and communication
      for Gluster. The problem is that I will have traffic over public
      WAN for a geo replication instance and I cannot go forward until
      this is solved.

    Was there a bug filed about this? I've looked at the issues on
      the Glusterfs github page but I didn't find anything related to
      this:

    https://github.com/gluster/glusterfs/issues

    Regards,
    Andrei

    On 7/12/18 4:03 PM, David Spisla wrote:

        Hello Andrei,

        I am also using Gluster 4.1 on CentOS and I have the same
          problem. I tested it with a volume which had no network
          encryption and one with network encryption. You are not the
          only one.
        It seems to be a bug. At the moment there is no other
          choice to disable client.ssl and sever.ssl on a volume to have
          stable I/O.

        Regards
        David

        2018-07-12 10:45 GMT+02:00 Havriliuc
          Andrei <andrei@xxxxxxxxxxxxx>:

          Hello,

            I am doing some tests with GlusterFS 4.0 and I can't seem to
            solve some SSL/TLS issues. I am trying to set up a 2 node
            replicated gluster volume with SSL/TLS. For this setup, I
            use 3 KVM VMs (2 storage nodes + 1 client node). For the
            networking part, I use a dedicated private LAN for the KVM
            VMs. Each VM is able to ping the other, so there's no
            problem with the connectivity.

            To try to make the procedure I used as clear as possible, I
            will put all commands in chronological order:

            =====================================================

            1. First, I update the systems, install ntp and then reboot:

            yum update

            yum install ntp

            systemctl status ntpd

            systemctl start ntpd

            systemctl enable ntpd

            systemctl status ntpd

            =====================================================

            2. I use a separate partition within the two VM storage
            nodes. Each of the two nodes see the separate partition as
            /dev/sdb. After creating a thinly provisioned LV on each of
            the two nodes, I create an XFS filesystem on them:

            pvcreate /dev/sdb1

            vgcreate -s 32M vg_glusterfs /dev/sdb1

            lvcreate -L 20G --thinpool glusterfs_thin_pool vg_glusterfs

            lvcreate -V 15G --thin -n glusterfs_thin_vol1
            vg_glusterfs/glusterfs_thin_pool

            mkfs.xfs -i size=512 /dev/vg_glusterfs/glusterfs_thin_vol1

            =====================================================

            3. On each node, I configured the brick data partition and
            added the following to /etc/fstab:

            mkdir /data

            echo "/dev/vg_glusterfs/glusterfs_thin_vol1 /data xfs
            defaults 1 2" >> /etc/fstab

            mount -a

            =====================================================

            4. After mounting the volume, I see the following in df -Th,
            which is correct:

            [root@gluster1 brick1]# df -Th

            Filesystem                                   Type      Size 
            Used Avail Use% Mounted on

            /dev/sda1                                    ext4       46G 
            1.4G 42G   4% /

            devtmpfs                                     devtmpfs  3.9G 
               0 3.9G   0% /dev

            tmpfs                                        tmpfs     3.9G 
               0 3.9G   0% /dev/shm

            tmpfs                                        tmpfs     3.9G 
            8.6M 3.9G   1% /run

            tmpfs                                        tmpfs     3.9G 
               0 3.9G   0% /sys/fs/cgroup

            tmpfs                                        tmpfs     783M 
               0 783M   0% /run/user/0

            /dev/mapper/vg_glusterfs-glusterfs_thin_vol1 xfs       
            15G   34M 15G   1% /data

            =====================================================

            5. Create specific volume dirs on all storage nodes:

            mkdir -pv /data/glusterfs/${HOSTNAME%%.*}/vol01

            =====================================================

            6. Add entries in /etc/hosts:

            vim /etc/hosts

            192.168.10.233    gluster1

            192.168.10.234    gluster2

            192.168.10.237    gluster-client

            =====================================================

            7. Install gluster from the CentOS SIG:

            yum search centos-release-gluster

            yum install centos-release-gluster40

            yum install glusterfs-server

            =====================================================

            8. Set up TLS/SSL encryption on all nodes and clients
            (gluster1, gluster2, gluster-client):

            openssl genrsa -out /etc/ssl/glusterfs.key 2048

            In gluster1 node:

            openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj
            "/CN=gluster1" -out /etc/ssl/glusterfs.pem

            In gluster2 node:

            openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj
            "/CN=gluster2" -out /etc/ssl/glusterfs.pem

            In gluster-client node:

            openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj
            "/CN=gluster-client" -out /etc/ssl/glusterfs.pem

            =====================================================

            9. On another box, I concatenate all of the .pem
            certificates into a .ca file.

            Bring all .pem files locally:

            scp gluster1:/etc/ssl/glusterfs.pem
            gluster01.pem

            scp gluster2:/etc/ssl/glusterfs.pem
            gluster02.pem

            scp gluster-client:/etc/ssl/glusterfs.pem
            gluster-client.pem

            For storage nodes, I concatenate all .pem certificates
            (including the client's .pem):

            cat gluster01.pem gluster02.pem gluster-client.pem > glusterfs-nodes.ca

            For server clients:

            cat gluster01.pem gluster02.pem > glusterfs-client.ca

            =====================================================

            10. Put glusterfs-nodes.ca
            file on all the storage nodes (this includes storage nodes
            .pem + client's .pem):

            scp glusterfs-nodes.ca
            gluster1:/etc/ssl/glusterfs.ca

            scp glusterfs-nodes.ca
            gluster2:/etc/ssl/glusterfs.ca

            Put glusterfs-client.ca
            file on all the storage nodes (this includes only storage
            nodes .pem):

            scp glusterfs-client.ca
            gluster-client:/etc/ssl/glusterfs.ca

            =====================================================

            11. Enable management encryption on each node (command ran
            on gluster1, gluster2, gluster-client):

            touch /var/lib/glusterd/secure-access

            =====================================================

            12. Start, enable and check status of glusterd on gluster1
            and gluster2:

            systemctl start glusterd

            systemctl enable glusterd

            systemctl status glusterd

            =====================================================

            13. Configure trusted storage pool (TSP):

            >From gluster1:

            gluster peer status

            gluster peer probe gluster2

            =====================================================

            14. Create the replicated gluster volume but don't start it:

            gluster volume create vol01 replica 2 transport tcp
            gluster1:/data/glusterfs/gluster1/vol01/brick1
            gluster2:/data/glusterfs/gluster2/vol01/brick1

            =====================================================

            15. Setup SSL/TLS access to the volume:

            gluster volume set vol01 auth.ssl-allow
            'gluster01,gluster02,gluster-client'

            gluster volume set vol01 client.ssl on

            gluster volume set vol01 server.ssl on

            gluster volume set vol01 network.ping-timeout "5"

            gluster volume start vol01

            =====================================================

            16. Mount the volume on gluster-client:

            mount -t glusterfs gluster1:/vol01 /mnt

            After mounting the volume, df -h has the correct output:

            [root@gluster-client mnt]# df -h

            Filesystem       Size  Used Avail Use% Mounted on

            /dev/sda1         48G   36G  9.3G  80% /

            devtmpfs         1.9G     0  1.9G   0% /dev

            tmpfs            1.9G     0  1.9G   0% /dev/shm

            tmpfs            1.9G  8.5M  1.9G   1% /run

            tmpfs            1.9G     0  1.9G   0% /sys/fs/cgroup

            tmpfs            379M     0  379M   0% /run/user/0

            gluster1:/vol01   15G  187M   15G   2% /mnt

            =====================================================

            17. If I try to copy an archive, it throws an error after
            transferring part of the file and the /mnt folder becomes
            inaccessible for a while:

            [root@gluster-client mnt]# cp /root/mybee.tar.gz /mnt/

            cp: error writing ‘/mnt/mybee.tar.gz’: Transport endpoint is
            not connected

            cp: failed to extend ‘/mnt/mybee.tar.gz’: Transport endpoint
            is not connected

            cp: failed to close ‘/mnt/mybee.tar.gz’: Transport endpoint
            is not connected

            [root@gluster-client mnt]# ll /mnt

            ls: cannot open directory .: Transport endpoint is not
            connected

            =====================================================

            This is the log from gluster-client for the mnt mountpoint
            (mnt.log):

            [2018-06-29 07:52:03.869382] I [MSGID: 100030]
            [glusterfsd.c:2625:main] 0-/usr/sbin/glusterfs: Started
            running /usr/sbin/glusterfs version 4.0.2 (args:
            /usr/sbin/glusterfs --process-name fuse
            --volfile-server=gluster1 --volfile-id=/vol01 /mnt)

            [2018-06-29 07:52:03.878709] I [socket.c:4470:socket_init]
            0-glusterfs: SSL support on the I/O path is ENABLED

            [2018-06-29 07:52:03.878782] I [socket.c:4473:socket_init]
            0-glusterfs: SSL support for glusterd is ENABLED

            [2018-06-29 07:52:03.878795] I [socket.c:4490:socket_init]
            0-glusterfs: using private polling thread

            [2018-06-29 07:52:03.879108] E [socket.c:4541:socket_init]
            0-glusterfs: failed to open /etc/ssl/dhparam.pem, DH ciphers
            are disabled

            [2018-06-29 07:52:03.885718] I [MSGID: 101190]
            [event-epoll.c:609:event_dispatch_epoll_worker]
            0-epoll: Started thread with index 1

            [2018-06-29 07:52:03.901983] I [MSGID: 101190]
            [event-epoll.c:609:event_dispatch_epoll_worker]
            0-epoll: Started thread with index 2

            [2018-06-29 07:52:03.902092] I [socket.c:4470:socket_init]
            0-vol01-client-1: SSL support on the I/O path is ENABLED

            [2018-06-29 07:52:03.902117] I [socket.c:4473:socket_init]
            0-vol01-client-1: SSL support for glusterd is ENABLED

            [2018-06-29 07:52:03.902142] I [socket.c:4490:socket_init]
            0-vol01-client-1: using private polling thread

            [2018-06-29 07:52:03.902266] E [socket.c:4541:socket_init]
            0-vol01-client-1: failed to open /etc/ssl/dhparam.pem, DH
            ciphers are disabled

            [2018-06-29 07:52:03.902977] I [socket.c:4470:socket_init]
            0-vol01-client-0: SSL support on the I/O path is ENABLED

            [2018-06-29 07:52:03.903018] I [socket.c:4473:socket_init]
            0-vol01-client-0: SSL support for glusterd is ENABLED

            [2018-06-29 07:52:03.903028] I [socket.c:4490:socket_init]
            0-vol01-client-0: using private polling thread

            [2018-06-29 07:52:03.903161] E [socket.c:4541:socket_init]
            0-vol01-client-0: failed to open /etc/ssl/dhparam.pem, DH
            ciphers are disabled

            [2018-06-29 07:52:03.903756] I [MSGID: 114020]
            [client.c:2300:notify] 0-vol01-client-0: parent translators
            are ready, attempting connect on transport

            [2018-06-29 07:52:03.908432] I [MSGID: 114020]
            [client.c:2300:notify] 0-vol01-client-1: parent translators
            are ready, attempting connect on transport

            Final graph:

            +------------------------------------------------------------------------------+

              1: volume vol01-client-0

              2:     type protocol/client

              3:     option ping-timeout 5

              4:     option remote-host gluster1

              5:     option remote-subvolume
            /data/glusterfs/gluster1/vol01/brick1

              6:     option transport-type socket

              7:     option transport.address-family inet

              8:     option transport.socket.ssl-enabled on

              9:     option transport.tcp-user-timeout 0

             10:     option transport.socket.keepalive-time 20

             11:     option transport.socket.keepalive-interval 2

             12:     option transport.socket.keepalive-count 9

             13:     option send-gids true

             14: end-volume

             15:

             16: volume vol01-client-1

             17:     type protocol/client

             18:     option ping-timeout 5

             19:     option remote-host gluster2

             20:     option remote-subvolume
            /data/glusterfs/gluster2/vol01/brick1

             21:     option transport-type socket

             22:     option transport.address-family inet

             23:     option transport.socket.ssl-enabled on

             24:     option transport.tcp-user-timeout 0

             25:     option transport.socket.keepalive-time 20

             26:     option transport.socket.keepalive-interval 2

             27:     option transport.socket.keepalive-count 9

             28:     option send-gids true

             29: end-volume

             30:

             31: volume vol01-replicate-0

             32:     type cluster/replicate

             33:     option afr-pending-xattr
            vol01-client-0,vol01-client-1

             34:     option use-compound-fops off

             35:     subvolumes vol01-client-0 vol01-client-1

             36: end-volume

             37:

             38: volume vol01-dht

             39:     type cluster/distribute

             40:     option lock-migration off

             41:     option force-migration off

             42:     subvolumes vol01-replicate-0

             43: end-volume

             44:

             45: volume vol01-write-behind

             46:     type performance/write-behind

             47:     subvolumes vol01-dht

             48: end-volume

             49:

             50: volume vol01-read-ahead

             51:     type performance/read-ahead

             52:     subvolumes vol01-write-behind

             53: end-volume

             54:

             55: volume vol01-readdir-ahead

             56:     type performance/readdir-ahead

             57:     option parallel-readdir off

             58:     option rda-request-size 131072

             59:     option rda-cache-limit 10MB

             60:     subvolumes vol01-read-ahead

             61: end-volume

             62:

             63: volume vol01-io-cache

             64:     type performance/io-cache

             65:     subvolumes vol01-readdir-ahead

             66: end-volume

             67:

             68: volume vol01-quick-read

             69:     type performance/quick-read

             70:     subvolumes vol01-io-cache

             71: end-volume

             72:

             73: volume vol01-open-behind

             74:     type performance/open-behind

             75:     subvolumes vol01-quick-read

             76: end-volume

             77:

             78: volume vol01-md-cache

             79:     type performance/md-cache

             80:     subvolumes vol01-open-behind

             81: end-volume

             82:

             83: volume vol01

             84:     type debug/io-stats

             85:     option log-level INFO

             86:     option latency-measurement off

             87:     option count-fop-hits off

             88:     subvolumes vol01-md-cache

             89: end-volume

             90:

             91: volume meta-autoload

             92:     type meta

             93:     subvolumes vol01

             94: end-volume

             95:

            +------------------------------------------------------------------------------+

            [2018-06-29 07:52:03.917518] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:03.918459] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:03.919289] I
            [rpc-clnt.c:2071:rpc_clnt_reconfig] 0-vol01-client-0:
            changing port to 49152 (from 0)

            [2018-06-29 07:52:03.922903] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:03.923922] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:03.925115] I
            [rpc-clnt.c:2071:rpc_clnt_reconfig] 0-vol01-client-1:
            changing port to 49152 (from 0)

            [2018-06-29 07:52:03.930023] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:03.930756] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:03.931859] I [MSGID: 114046]
            [client-handshake.c:1176:client_setvolume_cbk]
            0-vol01-client-0: Connected to vol01-client-0, attached to
            remote volume '/data/glusterfs/gluster1/vol01/brick1'.

            [2018-06-29 07:52:03.931894] I [MSGID: 108005]
            [afr-common.c:5081:__afr_handle_child_up_event]
            0-vol01-replicate-0: Subvolume 'vol01-client-0' came back
            up; going online.

            [2018-06-29 07:52:03.938903] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:03.939803] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:03.941300] I [MSGID: 114046]
            [client-handshake.c:1176:client_setvolume_cbk]
            0-vol01-client-1: Connected to vol01-client-1, attached to
            remote volume '/data/glusterfs/gluster2/vol01/brick1'.

            [2018-06-29 07:52:03.942884] I
            [fuse-bridge.c:4234:fuse_init] 0-glusterfs-fuse: FUSE inited
            with protocol versions: glusterfs 7.24 kernel 7.22

            [2018-06-29 07:52:03.942907] I
            [fuse-bridge.c:4864:fuse_graph_sync] 0-fuse: switched
            to graph 0

            [2018-06-29 07:52:03.946767] W [socket.c:592:__socket_rwv]
            0-vol01-client-0: readv on 192.168.10.233:49152
            failed (Input/output error)

            [2018-06-29 07:52:03.946791] E [socket.c:2785:socket_poller]
            0-vol01-client-0: socket_poller 192.168.10.233:49152
            failed (Input/output error)

            [2018-06-29 07:52:03.946837] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:03.947264] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2018-06-29
            07:52:03.945958 (xid=0xb)

            [2018-06-29 07:52:03.947297] W [MSGID: 114031]
            [client-rpc-fops_v2.c:2540:client4_0_lookup_cbk]
            0-vol01-client-0: remote operation failed. Path: /
            (00000000-0000-0000-0000-000000000001) [Transport
            endpoint is not connected]

            [2018-06-29 07:52:03.947455] I [MSGID: 109005]
            [dht-selfheal.c:2328:dht_selfheal_directory]
            0-vol01-dht: Directory selfheal failed: Unable to form
            layout for directory /

            [2018-06-29 07:52:07.885389] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:07.886457] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:07.887452] I
            [rpc-clnt.c:2071:rpc_clnt_reconfig] 0-vol01-client-0:
            changing port to 49152 (from 0)

            [2018-06-29 07:52:07.900627] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:07.901594] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:07.903220] I [MSGID: 114046]
            [client-handshake.c:1176:client_setvolume_cbk]
            0-vol01-client-0: Connected to vol01-client-0, attached to
            remote volume '/data/glusterfs/gluster1/vol01/brick1'.

            [2018-06-29 07:52:13.062404] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:13.062780] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.048761 (xid=0x22)

            [2018-06-29 07:52:13.062826] W [MSGID: 114031]
            [client-rpc-fops_v2.c:658:client4_0_writev_cbk]
            0-vol01-client-0: remote operation failed [Transport
            endpoint is not connected]

            [2018-06-29 07:52:13.063017] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.049162 (xid=0x23)

            [2018-06-29 07:52:13.063175] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.049865 (xid=0x24)

            [2018-06-29 07:52:13.063319] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.050199 (xid=0x25)

            [2018-06-29 07:52:13.063473] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.050489 (xid=0x26)

            [2018-06-29 07:52:13.063600] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.050757 (xid=0x27)

            [2018-06-29 07:52:13.063745] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.051031 (xid=0x28)

            [2018-06-29 07:52:13.063883] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.053035 (xid=0x29)

            The message "W [MSGID: 114031]
            [client-rpc-fops_v2.c:658:client4_0_writev_cbk]
            0-vol01-client-0: remote operation failed [Transport
            endpoint is not connected]" repeated 7 times between
            [2018-06-29 07:52:13.062826] and [2018-06-29
            07:52:13.063903]

            [2018-06-29 07:52:13.104580] W [MSGID: 114031]
            [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk]
            0-vol01-client-0: remote operation failed

            [2018-06-29 07:52:13.109230] W [socket.c:592:__socket_rwv]
            0-vol01-client-1: writev on 192.168.10.234:49152
            failed (No data available)

            [2018-06-29 07:52:13.109282] E [socket.c:2777:socket_poller]
            0-vol01-client-1: poll error on socket

            [2018-06-29 07:52:13.109340] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:13.109357] E [MSGID: 108006]
            [afr-common.c:5158:__afr_handle_child_down_event]
            0-vol01-replicate-0: All subvolumes are down. Going offline
            until atleast one of them comes back up.

            [2018-06-29 07:52:13.109461] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up

            [2018-06-29 07:52:13.109679] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FSTAT(25)) called at 2018-06-29
            07:52:13.104513 (xid=0x42)

            [2018-06-29 07:52:13.109700] W [MSGID: 114031]
            [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk]
            0-vol01-client-1: remote operation failed [Transport
            endpoint is not connected]

            [2018-06-29 07:52:13.109854] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FXATTROP(34)) called at 2018-06-29
            07:52:13.104729 (xid=0x43)

            [2018-06-29 07:52:13.109877] W [MSGID: 114031]
            [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk]
            0-vol01-client-1: remote operation failed

            [2018-06-29 07:52:13.109897] E [MSGID: 114031]
            [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
            0-vol01-client-0: remote operation failed [Transport
            endpoint is not connected]

            [2018-06-29 07:52:13.110036] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.105623 (xid=0x44)

            [2018-06-29 07:52:13.110053] W [MSGID: 114031]
            [client-rpc-fops_v2.c:658:client4_0_writev_cbk]
            0-vol01-client-1: remote operation failed [Transport
            endpoint is not connected]

            [2018-06-29 07:52:13.110232] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FSTAT(25)) called at 2018-06-29
            07:52:13.105658 (xid=0x45)

            [2018-06-29 07:52:13.110250] W [MSGID: 114031]
            [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk]
            0-vol01-client-1: remote operation failed [Transport
            endpoint is not connected]

            [2018-06-29 07:52:13.110386] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FSTAT(25)) called at 2018-06-29
            07:52:13.106862 (xid=0x46)

            [2018-06-29 07:52:13.110530] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FSTAT(25)) called at 2018-06-29
            07:52:13.108103 (xid=0x47)

            [2018-06-29 07:52:13.110660] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FSTAT(25)) called at 2018-06-29
            07:52:13.108969 (xid=0x48)

            [2018-06-29 07:52:13.110793] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FXATTROP(34)) called at 2018-06-29
            07:52:13.109256 (xid=0x49)

            The message "W [MSGID: 114031]
            [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk]
            0-vol01-client-1: remote operation failed [Transport
            endpoint is not connected]" repeated 3 times between
            [2018-06-29 07:52:13.110250] and [2018-06-29
            07:52:13.110675]

            [2018-06-29 07:52:13.110808] W [MSGID: 114031]
            [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk]
            0-vol01-client-1: remote operation failed

            [2018-06-29 07:52:13.110933] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FXATTROP(34)) called at 2018-06-29
            07:52:13.109284 (xid=0x4a)

            The message "W [MSGID: 114031]
            [client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk]
            0-vol01-client-1: remote operation failed" repeated 3 times
            between [2018-06-29 07:52:13.110808] and [2018-06-29
            07:52:14.878874]

            [2018-06-29 07:52:14.878895] E [MSGID: 114031]
            [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
            0-vol01-client-1: remote operation failed [Transport
            endpoint is not connected]

            [2018-06-29 07:52:14.879095] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up

            [2018-06-29 07:52:14.879217] W
            [fuse-bridge.c:2427:fuse_writev_cbk] 0-glusterfs-fuse:
            76: WRITE => -1 gfid=a8848be6-1c05-4a72-88a7-5209eeef7c2c
            fd=0x7f7c50023a68 (Transport endpoint is not connected)

            [2018-06-29 07:52:14.880509] W
            [fuse-bridge.c:1406:fuse_err_cbk] 0-glusterfs-fuse: 77:
            FLUSH() ERR => -1 (Transport endpoint is not connected)

            The message "I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up" repeated 11 times between [2018-06-29
            07:52:14.879095] and [2018-06-29 07:52:17.908056]

            [2018-06-29 07:52:17.908066] E [MSGID: 101046]
            [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht:
            dict is null

            [2018-06-29 07:52:17.908175] W
            [fuse-bridge.c:896:fuse_attr_cbk] 0-glusterfs-fuse: 79:
            LOOKUP() / => -1 (Transport endpoint is not connected)

            [2018-06-29 07:52:18.758975] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up

            [2018-06-29 07:52:18.759155] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up

            [2018-06-29 07:52:18.759165] E [MSGID: 101046]
            [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht:
            dict is null

            [2018-06-29 07:52:18.759214] W
            [fuse-bridge.c:896:fuse_attr_cbk] 0-glusterfs-fuse: 80:
            LOOKUP() / => -1 (Transport endpoint is not connected)

            [2018-06-29 07:52:21.662626] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up

            [2018-06-29 07:52:21.662756] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up

            [2018-06-29 07:52:21.662765] E [MSGID: 101046]
            [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht:
            dict is null

            [2018-06-29 07:52:21.662811] W
            [fuse-bridge.c:896:fuse_attr_cbk] 0-glusterfs-fuse: 81:
            LOOKUP() / => -1 (Transport endpoint is not connected)

            [2018-06-29 07:52:21.663567] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up

            [2018-06-29 07:52:21.663630] W
            [fuse-bridge.c:3205:fuse_statfs_cbk] 0-glusterfs-fuse:
            82: ERR => -1 (Transport endpoint is not connected)

            [2018-06-29 07:52:23.896362] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:23.896687] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:23.897321] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:23.896427 (xid=0x2a)

            [2018-06-29 07:52:23.897353] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:23.903241] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:23.903566] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:23.904166] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:23.903281 (xid=0x4b)

            [2018-06-29 07:52:23.904199] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:26.909819] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:26.910175] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:26.910858] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:26.909871 (xid=0x2b)

            [2018-06-29 07:52:26.910888] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:26.913921] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:26.914163] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:26.914417] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:26.913961 (xid=0x4c)

            [2018-06-29 07:52:26.914443] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:29.921628] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:29.921867] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:29.922215] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:29.921674 (xid=0x2c)

            [2018-06-29 07:52:29.922257] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:29.926462] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:29.926774] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:29.927415] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:29.926499 (xid=0x4d)

            [2018-06-29 07:52:29.927447] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:32.935218] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:32.935553] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:32.936196] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:32.935271 (xid=0x2d)

            [2018-06-29 07:52:32.936230] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:32.938940] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:32.939232] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:32.939497] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:32.938982 (xid=0x4e)

            [2018-06-29 07:52:32.939526] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:35.948789] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:35.949172] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:35.949860] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:35.948839 (xid=0x2e)

            [2018-06-29 07:52:35.949901] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:35.952154] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:35.952408] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:35.952670] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:35.952199 (xid=0x4f)

            [2018-06-29 07:52:35.952699] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:38.959369] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:38.959733] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:38.960536] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:38.959426 (xid=0x2f)

            [2018-06-29 07:52:38.960573] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:38.963381] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:38.963618] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:38.963877] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:38.963428 (xid=0x50)

            [2018-06-29 07:52:38.963906] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:41.969152] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:41.969397] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:41.969731] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:41.969204 (xid=0x30)

            [2018-06-29 07:52:41.969769] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:41.974130] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:41.974381] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:41.974669] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:41.974198 (xid=0x51)

            [2018-06-29 07:52:41.974698] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:44.984465] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:44.984781] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:44.985444] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:44.984514 (xid=0x31)

            [2018-06-29 07:52:44.985477] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:44.990039] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:44.990352] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:44.990729] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:44.990094 (xid=0x52)

            [2018-06-29 07:52:44.990771] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:47.994448] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:47.994684] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:47.994947] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:47.994499 (xid=0x32)

            [2018-06-29 07:52:47.994984] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:48.002544] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:48.002809] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:48.003388] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:48.002577 (xid=0x53)

            [2018-06-29 07:52:48.003416] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:49.440101] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up

            [2018-06-29 07:52:49.440257] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up

            [2018-06-29 07:52:49.440266] E [MSGID: 101046]
            [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht:
            dict is null

            [2018-06-29 07:52:49.440310] W
            [fuse-bridge.c:896:fuse_attr_cbk] 0-glusterfs-fuse: 83:
            LOOKUP() / => -1 (Transport endpoint is not connected)

            [2018-06-29 07:52:50.182948] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up

            [2018-06-29 07:52:50.183108] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up

            [2018-06-29 07:52:50.183116] E [MSGID: 101046]
            [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht:
            dict is null

            [2018-06-29 07:52:50.197572] W
            [fuse-bridge.c:896:fuse_attr_cbk] 0-glusterfs-fuse: 84:
            LOOKUP() / => -1 (Transport endpoint is not connected)

            [2018-06-29 07:52:50.575887] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up

            [2018-06-29 07:52:50.576030] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_init] 0-vol01-replicate-0:
            no subvolumes up

            [2018-06-29 07:52:50.576039] E [MSGID: 101046]
            [dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht:
            dict is null

            [2018-06-29 07:52:50.576087] W
            [fuse-bridge.c:896:fuse_attr_cbk] 0-glusterfs-fuse: 85:
            LOOKUP() / => -1 (Transport endpoint is not connected)

            [2018-06-29 07:52:51.012626] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:51.012924] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:51.013615] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:51.012667 (xid=0x33)

            [2018-06-29 07:52:51.013647] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]

            [2018-06-29 07:52:51.018855] W
            [rpc-clnt.c:1739:rpc_clnt_submit] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0

            [2018-06-29 07:52:51.019093] I [MSGID: 114018]
            [client.c:2227:client_rpc_notify] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick's port is
            available

            [2018-06-29 07:52:51.019378] E
            [rpc-clnt.c:350:saved_frames_unwind] (-->
            /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b]
            (--> /lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e]
            (--> /lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e]
            (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d]
            (--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:51.018893 (xid=0x54)

            [2018-06-29 07:52:51.019408] W [MSGID: 114032]
            [client-handshake.c:1620:client_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]

            =====================================================

            After trying to copy the archive, I end up with different
            sized files on the two bricks:

            [root@gluster1 brick1]# pwd

            /data/glusterfs/gluster1/vol01/brick1

            [root@gluster1 brick1]# ll

            total 384

            -rw------- 2 root root 393216 Jun 29 10:52 mybee.tar.gz

            [root@gluster2 brick1]# pwd

            /data/glusterfs/gluster2/vol01/brick1

            [root@gluster2 brick1]# ll

            total 1280

            -rw------- 2 root root 1310720 Jun 29 10:52 mybee.tar.gz

            =====================================================

            These are the informations regarding the gluster volume:

            [root@gluster1 glusterfs]# gluster volume info vol01

            Volume Name: vol01

            Type: Replicate

            Volume ID: d3b39520-3fbb-4cfc-b736-338d7726b5a1

            Status: Started

            Snapshot Count: 0

            Number of Bricks: 1 x 2 = 2

            Transport-type: tcp

            Bricks:

            Brick1: gluster1:/data/glusterfs/gluster1/vol01/brick1

            Brick2: gluster2:/data/glusterfs/gluster2/vol01/brick1

            Options Reconfigured:

            network.ping-timeout: 5

            server.ssl: on

            client.ssl: on

            auth.ssl-allow: gluster01,gluster02,gluster-client

            transport.address-family: inet

            nfs.disable: on

            performance.client-io-threads: off

            =====================================================

            Here is the peers information:

            [root@gluster1 brick1]# gluster peer status

            Number of Peers: 1

            Hostname: gluster2

            Uuid: 9ccac8e3-71c1-43e4-b723-fb7374be8a1d

            State: Peer in Cluster (Connected)

            =====================================================

            Here is the volume status:

            [root@gluster1 brick1]# gluster volume status vol01

            Status of volume: vol01

            Gluster process                             TCP Port  RDMA
            Port Online  Pid

            ------------------------------------------------------------------------------

            Brick gluster1:/data/glusterfs/gluster1/vol

            01/brick1                                   49152     0 Y   
               11108

            Brick gluster2:/data/glusterfs/gluster2/vol

            01/brick1                                   49152     0 Y   
               10942

            Self-heal Daemon on localhost               N/A       N/A Y 
                 11133

            Self-heal Daemon on gluster2                N/A       N/A Y 
                 10968

            Task Status of Volume vol01

            ------------------------------------------------------------------------------

            There are no active volume tasks

            =====================================================

            These are the installed packages on gluster-client:

            [root@gluster-client glusterfs]# rpm -qa | grep
            "gluster\|fuse"

            centos-release-gluster40-1.0-2.el7.centos.x86_64

            glusterfs-client-xlators-4.0.2-1.el7.x86_64

            fuse-2.9.2-10.el7.x86_64

            glusterfs-4.0.2-1.el7.x86_64

            glusterfs-fuse-4.0.2-1.el7.x86_64

            fuse-libs-2.9.2-10.el7.x86_64

            glusterfs-libs-4.0.2-1.el7.x86_64

            And these are the installed packages on gluster1 and
            gluster2 storage nodes:

            [root@gluster1 glusterfs]# rpm -qa | grep "gluster\|fuse"

            glusterfs-api-4.0.2-1.el7.x86_64

            centos-release-gluster40-1.0-2.el7.centos.x86_64

            glusterfs-libs-4.0.2-1.el7.x86_64

            glusterfs-client-xlators-4.0.2-1.el7.x86_64

            glusterfs-cli-4.0.2-1.el7.x86_64

            glusterfs-fuse-4.0.2-1.el7.x86_64

            glusterfs-server-4.0.2-1.el7.x86_64

            glusterfs-4.0.2-1.el7.x86_64

            Can anyone point me in the right direction? What am I doing
            wrong? The mount works fine until I try to copy an archive,
            multiple smaller files or a bigger file on it (meaning it
            shows correctly in df -Th and I can create several files
            with "touch file1 file2..."). Basically, after any data
            transfer, I get these errors.

            I followed the indications from the redhat page:

            https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/chap-network_encryption

            UPDATE 1:

            I tried doing the exact same steps in Gluster 3.12 and had
            no problem. The steps worked and SSL/TLS was enabled. There
            was no transport error or anything and I also checked if
            SSL/TLS was enabled. Afterwards, I also tried with the new
            release 4.1 and the problem persists (same error with
            "Transport endpoint is not connected").

            Let me know if you need any other info. Any help is much
            appreciated.

            Thank you.

            _______________________________________________

            Gluster-users mailing list

            Gluster-users@xxxxxxxxxxx

            https://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________

Gluster-users mailing list

Gluster-users@xxxxxxxxxxx

https://lists.gluster.org/mailman/listinfo/gluster-users

-- 
Amar Tumballi (amarts)

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users