Re: Transport Endpoint Not Connected When Writing a Lot of Files

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thank you for your response!

it does not reply.
but all other servers neither reply to ping <IP> 24007
pinged this from each server to each server, no reply at all.
even with firewall disabled it does not reply.

root@diufnas22:/home/diuf-sysadmin# netstat -tulpe |grep 24007
tcp 0 0 *:24007 *:* LISTEN root 47716960 141881/glusterd

Firewall:
8458,24007,24008,49150,49151,49152,49153,49154,49155,49156,49157,49158/tcp (v6) on bond1 ALLOW Anywhere (v6) 8458,24007,24008,49150,49151,49152,49153,49154,49155,49156,49157,49158/tcp (v6) on bond0 ALLOW Anywhere (v6)

Kind regards,
Birgit


On 16/10/19 05:59, Amar Tumballi wrote:
Went through the design. In my opinion, this makes sense, ie, as long as you can use better faster network as alternative to reach server, it is fine. And considering gluster servers are stateless, that shouldn't cause a problem.

Coming to the my suspicion of an issue of network in this particular issue comes from the log which mentions 'No route to host'  in the log. That particular log prints errno from RPC layer's connect() system call.

Snippet from `man 3 connect`

            EHOSTUNREACH
                   The destination host cannot be reached (probably
    because the  host  is  down  or  a  remote
                   router cannot reach it).


It may be working from few servers, but on the ones you are getting this error, can you check if you can reach the server (specific IP), using 'ping' or 'telnet <IP> 24007' ?

Regards,
Amar




On Tue, Oct 15, 2019 at 5:00 PM DUCARROZ Birgit <birgit.ducarroz@xxxxxxxx <mailto:birgit.ducarroz@xxxxxxxx>> wrote:

    Hi,

    I send you this mail without sending it to the gluster mailing list
    because of the pdf attachment (I do not want it to be published).

    Do you think it might be because of the different IP address of
    diufnas22 where the arbiter brick 3 is installed?

    2 hosts communicate with internal 192.168.x.x Network whereas the 3rd
    host diufnas22 is connected to the two other hosts via a switch, using
    ip address 134.21.x.x. Both networks have a speed of 20gb (2 x 10gb
    using lacp bond).

    (The scheme shows this).

    I would be able to remove the 192.168.x.x network, but my aim was to
    speed up the network using an internal communication between bricks
    1 and 2.

    If this is really the problem, why does the installation mostly works
    and crush down when there is heavy network usage with a lot of small
    files?


    Kind regards
-- Birgit Ducarroz
    Unix Systems Administration
    Department of Informatics
    University of Fribourg Switzerland
    mailto:birgit.ducarroz@xxxxxxxx <mailto:birgit.ducarroz@xxxxxxxx>
    Phone: +41 (26) 300 8342
    https://diuf.unifr.ch/people/ducarroz/
    INTRANET / SECURITY NEWS: https://diuf-file.unifr.ch

    On 14/10/19 12:47, Amar Tumballi wrote:
     > One of the host ( 134.21.57.122 <http://134.21.57.122:24007/>) is
    not
     > reachable from your network. Also checking at the IP, it would have
     > gotten resolved to something else than expected. Can you check if
     > 'diufnas22' is properly resolved?
     >
     > -Amar
     >
     > On Mon, Oct 14, 2019 at 3:44 PM DUCARROZ Birgit
     > <birgit.ducarroz@xxxxxxxx <mailto:birgit.ducarroz@xxxxxxxx>
    <mailto:birgit.ducarroz@xxxxxxxx <mailto:birgit.ducarroz@xxxxxxxx>>>
    wrote:
     >
     >     Thank you.
     >     I checked the logs but the information was not clear to me.
     >
     >     I add the log of two different crashes. I will do an upgrade to
     >     glusterFS 6 in some weeks. Actually I cannot interrupt user
    activity on
     >     these servers since we are in the middle of the uni-semester.
     >
     >     If these logfiles reveal something interesting to you, would
    be nice to
     >     get a hint.
     >
     >
     >     ol-data-client-2. Client process will keep trying to connect to
     >     glusterd
     >     until brick's port is available
     >     [2019-09-16 19:05:34.028164] E
    [rpc-clnt.c:348:saved_frames_unwind]
     >     (-->
>  /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7ff167753ddb]
     >
     >     (-->
    /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xc021)[0x7ff167523021]
     >     (-->
    /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xc14e)[0x7ff16752314e]
     >     (-->
>  /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8e)[0x7ff1675246be]
     >
     >     (-->
    /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe268)[0x7ff167525268]
     >     ))))) 0-vol-data-client-2: forced unwinding frame
    type(GlusterFS 4.x
     >     v1)
     >     op(FSTAT(25)) called at 2019-09-16 19:05:28.736873
    (xid=0x113aecf)
     >     [2019-09-16 19:05:34.028206] W [MSGID: 114031]
     >     [client-rpc-fops_v2.c:1260:client4_0_fstat_cbk]
    0-vol-data-client-2:
     >     remote operation failed [Transport endpoint is not connected]
     >     [2019-09-16 19:05:44.970828] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-09-16 19:05:44.971030] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-09-16 19:05:44.971165] E [MSGID: 114058]
     >     [client-handshake.c:1442:client_query_portmap_cbk]
    0-vol-data-client-2:
     >     failed to get the port number for remote subvolume. Please
    run 'gluster
     >     volume status' on server to see if brick process is running.
     >     [2019-09-16 19:05:47.971375] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >
     >     [2019-09-16 19:05:44.971200] I [MSGID: 114018]
     >     [client.c:2254:client_rpc_notify] 0-vol-data-client-2:
    disconnected
     >     from
     >     vol-data-client-2. Client process will keep trying to connect to
     >     glusterd until brick's port is available
     >
     >
     >
     >     [2019-09-17 07:43:44.807182] E [MSGID: 114058]
     >     [client-handshake.c:1442:client_query_portmap_cbk]
    0-vol-data-client-0:
     >     failed to get the port number for remote subvolume. Please
    run 'gluster
     >     volume status' on server to see if brick process is running.
     >     [2019-09-17 07:43:44.807217] I [MSGID: 114018]
     >     [client.c:2254:client_rpc_notify] 0-vol-data-client-0:
    disconnected
     >     from
     >     vol-data-client-0. Client process will keep trying to connect to
     >     glusterd until brick's port is available
     >     [2019-09-17 07:43:44.807228] E [MSGID: 108006]
     >     [afr-common.c:5413:__afr_handle_child_down_event]
     >     0-vol-data-replicate-0: All subvolumes are down. Going
    offline until
     >     atleast one of them comes back up.
     >     Final graph:
>  +------------------------------------------------------------------------------+
     >         1: volume vol-data-client-0
     >         2:     type protocol/client
     >         3:     option ping-timeout 42
     >         4:     option remote-host diufnas20
     >         5:     option remote-subvolume /bigdisk/brick1/vol-data
     >         6:     option transport-type socket
     >         7:     option transport.address-family inet
     >         8:     option username a14ffa1b-b64e-410c-894d-435c18e81b2d
     >         9:     option password 37ba4281-166d-40fd-9ef0-08a187d1107b
     >        10:     option transport.tcp-user-timeout 0
     >        11:     option transport.socket.keepalive-time 20
     >        12:     option transport.socket.keepalive-interval 2
     >        13:     option transport.socket.keepalive-count 9
     >        14:     option send-gids true
     >        15: end-volume
     >        16:
     >        17: volume vol-data-client-1
     >        18:     type protocol/client
     >        19:     option ping-timeout 42
     >        20:     option remote-host diufnas21
     >        21:     option remote-subvolume /bigdisk/brick2/vol-data
     >        22:     option transport-type socket
     >        23:     option transport.address-family inet
     >        24:     option username a14ffa1b-b64e-410c-894d-435c18e81b2d
     >        25:     option password 37ba4281-166d-40fd-9ef0-08a187d1107b
     >        26:     option transport.tcp-user-timeout 0
     >        27:     option transport.socket.keepalive-time 20
     >     29:     option transport.socket.keepalive-count 9
     >        30:     option send-gids true
     >        31: end-volume
     >        32:
     >        33: volume vol-data-client-2
     >        34:     type protocol/client
     >        35:     option ping-timeout 42
     >        36:     option remote-host diufnas22
     >        37:     option remote-subvolume /bigdisk/brick3/vol-data
     >        38:     option transport-type socket
     >        39:     option transport.address-family inet
     >        40:     option username a14ffa1b-b64e-410c-894d-435c18e81b2d
     >        41:     option password 37ba4281-166d-40fd-9ef0-08a187d1107b
     >        42:     option transport.tcp-user-timeout 0
     >        43:     option transport.socket.keepalive-time 20
     >        44:     option transport.socket.keepalive-interval 2
     >        45:     option transport.socket.keepalive-count 9
     >        46:     option send-gids true
     >        47: end-volume
     >        48:
     >     49: volume vol-data-replicate-0
     >        50:     type cluster/replicate
     >        51:     option afr-pending-xattr
     >     vol-data-client-0,vol-data-client-1,vol-data-client-2
     >        52:     option arbiter-count 1
     >        53:     option use-compound-fops off
     >        54:     subvolumes vol-data-client-0 vol-data-client-1
     >     vol-data-client-2
     >        55: end-volume
     >        56:
     >        57: volume vol-data-dht
     >        58:     type cluster/distribute
     >        59:     option min-free-disk 10%
     >        60:     option lock-migration off
     >        61:     option force-migration off
     >        62:     subvolumes vol-data-replicate-0
     >        63: end-volume
     >        64:
     >        65: volume vol-data-write-behind
     >        66:     type performance/write-behind
     >        67:     subvolumes vol-data-dht
     >        68: end-volume
     >        69:
     >        70: volume vol-data-read-ahead
     >        71:     type performance/read-ahead
     >        72:     subvolumes vol-data-write-behind
     >        73: end-volume
     >        74:
     >        75: volume vol-data-readdir-ahead
     >        76:     type performance/readdir-ahead
     >        77:     option parallel-readdir off
     >        78:     option rda-request-size 131072
     >        79:     option rda-cache-limit 10MB
     >        80:     subvolumes vol-data-read-ahead
     >        81: end-volume
     >        82:
     >        83: volume vol-data-io-cache
     >        84:     type performance/io-cache
     >        85:     option max-file-size 256MB
     >        86:     option cache-size 28GB
     >        87:     subvolumes vol-data-readdir-ahead
     >        88: end-volume
     >        89:
     >        90: volume vol-data-quick-read
     >        91:     type performance/quick-read
     >        92:     option cache-size 28GB
     >        93:     subvolumes vol-data-io-cache
     >        94: end-volume
     >        95:
     >        96: volume vol-data-open-behind
     >        97:     type performance/open-behind
     >        98:     subvolumes vol-data-quick-read
     >        99: end-volume
     >     100:
     >     101: volume vol-data-md-cache
     >     102:     type performance/md-cache
     >     103:     subvolumes vol-data-open-behind
     >     104: end-volume
     >     105:
     >     106: volume vol-data-io-threads
     >     107:     type performance/io-threads
     >     108:     subvolumes vol-data-md-cache
     >     109: end-volume
     >     110:
     >     111: volume vol-data
     >     112:     type debug/io-stats
     >     113:     option log-level INFO
     >     114:     option latency-measurement off
     >     115:     option count-fop-hits off
     >     116:     subvolumes vol-data-io-threads
     >     117: end-volume
     >     118:
     >     119: volume meta-autoload
     >     120:     type meta
     >     121:     subvolumes vol-data
     >     122: end-volume
     >     123:
>  +------------------------------------------------------------------------------+
     >     [2019-09-17 07:43:47.249546] E
    [socket.c:2524:socket_connect_finish]
     >     0-vol-data-client-2: connection to 134.21.57.122:24007
    <http://134.21.57.122:24007>
     >     <http://134.21.57.122:24007> failed (No route
     >     to host); disconnecting socket
     >     [2019-09-17 07:43:48.801700] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >
     >
     >
     >     root@nas20:/var/log/glusterfs# dmesg |grep error
     >     [    2.463658] i8042: probe of i8042 failed with error -5
     >     [    8.180404] EXT4-fs (sdb1): re-mounted. Opts:
    errors=remount-ro
     >     [   10.024111] EXT4-fs (sda): mounted filesystem with ordered
    data
     >     mode.
     >     Opts: errors=remount-ro
     >     [   64.432042] ureadahead[1478]: segfault at 7f4b99d3d2c0 ip
     >     00005629096fe2d1 sp 00007fff9dc98250 error 6 in
     >     ureadahead[5629096fa000+8000]
     >
     >
     >     root@nas20:/var/log/glusterfs# cat export-users.log | grep
     >     "2019-10-08 20"
     >     [2019-10-08 20:10:33.695082] I [MSGID: 100030]
    [glusterfsd.c:2741:main]
     >     0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs
    version
     >     4.1.8
     >     (args: /usr/sbin/glusterfs --process-name fuse
     >     --volfile-server=localhost --volfile-id=/vol-users /export/users)
     >     [2019-10-08 20:10:33.712430] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 1
     >     [2019-10-08 20:10:33.816594] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 2
     >     [2019-10-08 20:10:33.820975] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-users-client-0: parent translators are ready,
    attempting connect
     >     on transport
     >     [2019-10-08 20:10:33.821257] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-users-client-1: parent translators are ready,
    attempting connect
     >     on transport
     >     [2019-10-08 20:10:33.821466] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-users-client-2: parent translators are ready,
    attempting connect
     >     on transport
     >     [2019-10-08 20:10:33.822271] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:33.822425] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:33.822484] E [MSGID: 114058]
     >     [client-handshake.c:1442:client_query_portmap_cbk]
     >     0-vol-users-client-0:
     >     failed to get the port number for remote subvolume. Please
    run 'gluster
     >     volume status' on server to see if brick process is running.
     >     [2019-10-08 20:10:33.822518] I [MSGID: 114018]
     >     [client.c:2254:client_rpc_notify] 0-vol-users-client-0:
    disconnected
     >     from vol-users-client-0. Client process will keep trying to
    connect to
     >     glusterd until brick's port is available
     >     [2019-10-08 20:10:33.822528] E [MSGID: 108006]
     >     [afr-common.c:5413:__afr_handle_child_down_event]
     >     0-vol-users-replicate-0: All subvolumes are down. Going
    offline until
     >     atleast one of them comes back up.
     >     [2019-10-08 20:10:36.387074] E
    [socket.c:2524:socket_connect_finish]
     >     0-vol-users-client-2: connection to 134.21.57.122:24007
    <http://134.21.57.122:24007>
     >     <http://134.21.57.122:24007> failed (No route
     >     to host); disconnecting socket
     >     [2019-10-08 20:10:36.387120] E
    [socket.c:2524:socket_connect_finish]
     >     0-vol-users-client-1: connection to 192.168.1.121:24007
    <http://192.168.1.121:24007>
     >     <http://192.168.1.121:24007> failed (No route
     >     to host); disconnecting socket
     >     [2019-10-08 20:10:36.388236] I [fuse-bridge.c:4294:fuse_init]
     >     0-glusterfs-fuse: FUSE inited with protocol versions:
    glusterfs 7.24
     >     kernel 7.23
     >     [2019-10-08 20:10:36.388254] I
    [fuse-bridge.c:4927:fuse_graph_sync]
     >     0-fuse: switched to graph 0
     >     The message "E [MSGID: 108006]
     >     [afr-common.c:5413:__afr_handle_child_down_event]
     >     0-vol-users-replicate-0: All subvolumes are down. Going
    offline until
     >     atleast one of them comes back up." repeated 2 times between
     >     [2019-10-08
     >     20:10:33.822528] and [2019-10-08 20:10:36.387272]
     >     [2019-10-08 20:10:36.388596] I [MSGID: 108006]
     >     [afr-common.c:5677:afr_local_init] 0-vol-users-replicate-0: no
     >     subvolumes up
     >     [2019-10-08 20:10:36.388667] E [MSGID: 101046]
     >     [dht-common.c:1502:dht_lookup_dir_cbk] 0-vol-users-dht: dict
    is null
     >     [2019-10-08 20:10:36.388724] E
    [fuse-bridge.c:4362:fuse_first_lookup]
     >     0-fuse: first lookup on root failed (Transport endpoint is not
     >     connected)
     >     [2019-10-08 20:10:36.388847] I [MSGID: 108006]
     >     [afr-common.c:5677:afr_local_init] 0-vol-users-replicate-0: no
     >     subvolumes up
     >     [2019-10-08 20:10:36.388864] E [MSGID: 101046]
     >     [dht-common.c:1502:dht_lookup_dir_cbk] 0-vol-users-dht: dict
    is null
     >     [2019-10-08 20:10:36.388883] W
     >     [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
     >     00000000-0000-0000-0000-000000000001: failed to resolve
    (Transport
     >     endpoint is not connected)
     >     [2019-10-08 20:10:36.388893] E
    [fuse-bridge.c:928:fuse_getattr_resume]
     >     0-glusterfs-fuse: 2: GETATTR 1
    (00000000-0000-0000-0000-000000000001)
     >     resolution failed
     >     [2019-10-08 20:10:36.391191] I [MSGID: 108006]
     >     [afr-common.c:5677:afr_local_init] 0-vol-users-replicate-0: no
     >     subvolumes up
     >     [2019-10-08 20:10:36.391218] E [MSGID: 101046]
     >     [dht-common.c:1502:dht_lookup_dir_cbk] 0-vol-users-dht: dict
    is null
     >     [2019-10-08 20:10:36.391241] W
     >     [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
     >     00000000-0000-0000-0000-000000000001: failed to resolve
    (Transport
     >     endpoint is not connected)
     >     [2019-10-08 20:10:36.391250] E
    [fuse-bridge.c:928:fuse_getattr_resume]
     >     0-glusterfs-fuse: 3: GETATTR 1
    (00000000-0000-0000-0000-000000000001)
     >     resolution failed
     >     [2019-10-08 20:10:36.391317] I [MSGID: 108006]
     >     [afr-common.c:5677:afr_local_init] 0-vol-users-replicate-0: no
     >     subvolumes up
     >     [2019-10-08 20:10:36.391333] E [MSGID: 101046]
     >     [dht-common.c:1502:dht_lookup_dir_cbk] 0-vol-users-dht: dict
    is null
     >     [2019-10-08 20:10:36.391352] W
     >     [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
     >     00000000-0000-0000-0000-000000000001: failed to resolve
    (Transport
     >     endpoint is not connected)
     >     [2019-10-08 20:10:36.391360] E
    [fuse-bridge.c:928:fuse_getattr_resume]
     >     0-glusterfs-fuse: 4: GETATTR 1
    (00000000-0000-0000-0000-000000000001)
     >     resolution failed
     >     [2019-10-08 20:10:36.406967] I
    [fuse-bridge.c:5199:fuse_thread_proc]
     >     0-fuse: initating unmount of /export/users
     >     [2019-10-08 20:10:36.407298] W
    [glusterfsd.c:1514:cleanup_and_exit]
     >     (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)
    [0x7f88cc59b6ba]
     >     -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xed) [0x55c01427f70d]
     >     -->/usr/sbin/glusterfs(cleanup_and_exit+0x54)
    [0x55c01427f524] ) 0-:
     >     received signum (15), shutting down
     >     [2019-10-08 20:10:36.407318] I [fuse-bridge.c:5981:fini] 0-fuse:
     >     Unmounting '/export/users'.
     >     [2019-10-08 20:10:36.407326] I [fuse-bridge.c:5986:fini] 0-fuse:
     >     Closing
     >     fuse connection to '/export/users'.
     >     [2019-10-08 20:10:43.925719] I [MSGID: 100030]
    [glusterfsd.c:2741:main]
     >     0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs
    version
     >     4.1.8
     >     (args: /usr/sbin/glusterfs --process-name fuse
     >     --volfile-server=localhost --volfile-id=/vol-users /export/users)
     >     [2019-10-08 20:10:43.929529] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 1
     >     [2019-10-08 20:10:43.933210] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 2
     >     [2019-10-08 20:10:43.933789] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-users-client-0: parent translators are ready,
    attempting connect
     >     on transport
     >     [2019-10-08 20:10:43.934151] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-users-client-1: parent translators are ready,
    attempting connect
     >     on transport
     >     [2019-10-08 20:10:43.934174] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:43.934269] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:43.934331] E [MSGID: 114058]
     >     [client-handshake.c:1442:client_query_portmap_cbk]
     >     0-vol-users-client-0:
     >     failed to get the port number for remote subvolume. Please
    run 'gluster
     >     volume status' on server to see if brick process is running.
     >     [2019-10-08 20:10:43.934369] I [MSGID: 114018]
     >     [client.c:2254:client_rpc_notify] 0-vol-users-client-0:
    disconnected
     >     from vol-users-client-0. Client process will keep trying to
    connect to
     >     glusterd until brick's port is available
     >     [2019-10-08 20:10:43.934379] E [MSGID: 108006]
     >     [afr-common.c:5413:__afr_handle_child_down_event]
     >     0-vol-users-replicate-0: All subvolumes are down. Going
    offline until
     >     atleast one of them comes back up.
     >     [2019-10-08 20:10:43.934434] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-users-client-2: parent translators are ready,
    attempting connect
     >     on transport
     >     [2019-10-08 20:10:43.934574] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:43.934782] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:43.934859] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:43.934931] I
    [rpc-clnt.c:2105:rpc_clnt_reconfig]
     >     0-vol-users-client-1: changing port to 49154 (from 0)
     >     [2019-10-08 20:10:43.935152] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:43.935286] I
    [rpc-clnt.c:2105:rpc_clnt_reconfig]
     >     0-vol-users-client-2: changing port to 49154 (from 0)
     >     [2019-10-08 20:10:43.935314] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:43.935515] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:43.935711] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:43.935919] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:43.936354] I [MSGID: 114046]
     >     [client-handshake.c:1095:client_setvolume_cbk]
    0-vol-users-client-1:
     >     Connected to vol-users-client-1, attached to remote volume
     >     '/bigdisk/brick2/vol-users'.
     >     [2019-10-08 20:10:43.936375] I [MSGID: 108005]
     >     [afr-common.c:5336:__afr_handle_child_up_event]
     >     0-vol-users-replicate-0:
     >     Subvolume 'vol-users-client-1' came back up; going online.
     >     [2019-10-08 20:10:43.936728] I [MSGID: 114046]
     >     [client-handshake.c:1095:client_setvolume_cbk]
    0-vol-users-client-2:
     >     Connected to vol-users-client-2, attached to remote volume
     >     '/bigdisk/brick3/vol-users'.
     >     [2019-10-08 20:10:43.936742] I [MSGID: 108002]
     >     [afr-common.c:5611:afr_notify] 0-vol-users-replicate-0:
     >     Client-quorum is met
     >     [2019-10-08 20:10:43.937579] I [fuse-bridge.c:4294:fuse_init]
     >     0-glusterfs-fuse: FUSE inited with protocol versions:
    glusterfs 7.24
     >     kernel 7.23
     >     [2019-10-08 20:10:43.937595] I
    [fuse-bridge.c:4927:fuse_graph_sync]
     >     0-fuse: switched to graph 0
     >     [2019-10-08 20:10:43.939789] I [MSGID: 109005]
     >     [dht-selfheal.c:2342:dht_selfheal_directory] 0-vol-users-dht:
    Directory
     >     selfheal failed: Unable to form layout for directory /
     >     [2019-10-08 20:10:47.927439] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.927555] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.927627] I
    [rpc-clnt.c:2105:rpc_clnt_reconfig]
     >     0-vol-users-client-0: changing port to 49152 (from 0)
     >     [2019-10-08 20:10:47.928087] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.928201] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-users-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.928717] I [MSGID: 114046]
     >     [client-handshake.c:1095:client_setvolume_cbk]
    0-vol-users-client-0:
     >     Connected to vol-users-client-0, attached to remote volume
     >     '/bigdisk/brick1/vol-users'.
     >     root@nas20:/var/log/glusterfs# cat export-users.log | grep
     >     "2019-10-08 22"
     >     root@nas20:/var/log/glusterfs# cat export-users.log | grep
     >     "2019-10-08 21"
     >     root@nas20:/var/log/glusterfs# cat export-users.log | grep
     >     "2019-10-08 23"
     >     root@nas20:/var/log/glusterfs# cat export-data.log.log | grep
     >     "2019-10-08 23"
     >     cat: export-data.log.log: No such file or directory
     >     root@nas20:/var/log/glusterfs# cat export-data.log | grep
     >     "2019-10-08 15"
     >     root@nas20:/var/log/glusterfs# cat export-data.log | grep
     >     "2019-10-08 16"
     >     root@nas20:/var/log/glusterfs# cat export-data.log | grep
     >     "2019-10-08 17"
     >     root@nas20:/var/log/glusterfs# cat export-data.log | grep
     >     "2019-10-08 19"
     >     root@nas20:/var/log/glusterfs# cat export-data.log | grep
    "2019-10-08 1"
     >     root@nas20:/var/log/glusterfs# cat export-data.log | grep
     >     "2019-10-08 20"
     >     [2019-10-08 20:10:33.695000] I [MSGID: 100030]
    [glusterfsd.c:2741:main]
     >     0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs
    version
     >     4.1.8
     >     (args: /usr/sbin/glusterfs --process-name fuse
     >     --volfile-server=localhost --volfile-id=/vol-data /export/data)
     >     [2019-10-08 20:10:33.737302] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 1
     >     [2019-10-08 20:10:33.816578] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 2
     >     [2019-10-08 20:10:33.820946] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-0: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-08 20:10:33.821255] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-1: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-08 20:10:33.821467] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-2: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-08 20:10:33.822144] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:33.822243] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:33.822374] E [MSGID: 114058]
     >     [client-handshake.c:1442:client_query_portmap_cbk]
    0-vol-data-client-0:
     >     failed to get the port number for remote subvolume. Please
    run 'gluster
     >     volume status' on server to see if brick process is running.
     >     [2019-10-08 20:10:33.822412] I [MSGID: 114018]
     >     [client.c:2254:client_rpc_notify] 0-vol-data-client-0:
    disconnected
     >     from
     >     vol-data-client-0. Client process will keep trying to connect to
     >     glusterd until brick's port is available
     >     [2019-10-08 20:10:33.822423] E [MSGID: 108006]
     >     [afr-common.c:5413:__afr_handle_child_down_event]
     >     0-vol-data-replicate-0: All subvolumes are down. Going
    offline until
     >     atleast one of them comes back up.
     >     [2019-10-08 20:10:36.387062] E
    [socket.c:2524:socket_connect_finish]
     >     0-vol-data-client-2: connection to 134.21.57.122:24007
    <http://134.21.57.122:24007>
     >     <http://134.21.57.122:24007> failed (No route
     >     to host); disconnecting socket
     >     [2019-10-08 20:10:36.387091] E
    [socket.c:2524:socket_connect_finish]
     >     0-vol-data-client-1: connection to 192.168.1.121:24007
    <http://192.168.1.121:24007>
     >     <http://192.168.1.121:24007> failed (No route
     >     to host); disconnecting socket
     >     [2019-10-08 20:10:36.388218] I [fuse-bridge.c:4294:fuse_init]
     >     0-glusterfs-fuse: FUSE inited with protocol versions:
    glusterfs 7.24
     >     kernel 7.23
     >     [2019-10-08 20:10:36.388237] I
    [fuse-bridge.c:4927:fuse_graph_sync]
     >     0-fuse: switched to graph 0
     >     The message "E [MSGID: 108006]
     >     [afr-common.c:5413:__afr_handle_child_down_event]
     >     0-vol-data-replicate-0: All subvolumes are down. Going
    offline until
     >     atleast one of them comes back up." repeated 2 times between
     >     [2019-10-08
     >     20:10:33.822423] and [2019-10-08 20:10:36.387268]
     >     [2019-10-08 20:10:36.388590] I [MSGID: 108006]
     >     [afr-common.c:5677:afr_local_init] 0-vol-data-replicate-0: no
     >     subvolumes up
     >     [2019-10-08 20:10:36.388630] E [MSGID: 101046]
     >     [dht-common.c:1502:dht_lookup_dir_cbk] 0-vol-data-dht: dict
    is null
     >     [2019-10-08 20:10:36.388723] E
    [fuse-bridge.c:4362:fuse_first_lookup]
     >     0-fuse: first lookup on root failed (Transport endpoint is not
     >     connected)
     >     [2019-10-08 20:10:36.388855] I [MSGID: 108006]
     >     [afr-common.c:5677:afr_local_init] 0-vol-data-replicate-0: no
     >     subvolumes up
     >     [2019-10-08 20:10:36.388871] E [MSGID: 101046]
     >     [dht-common.c:1502:dht_lookup_dir_cbk] 0-vol-data-dht: dict
    is null
     >     [2019-10-08 20:10:36.388892] W
     >     [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
     >     00000000-0000-0000-0000-000000000001: failed to resolve
    (Transport
     >     endpoint is not connected)
     >     [2019-10-08 20:10:36.388902] E
    [fuse-bridge.c:928:fuse_getattr_resume]
     >     0-glusterfs-fuse: 2: GETATTR 1
    (00000000-0000-0000-0000-000000000001)
     >     resolution failed
     >     [2019-10-08 20:10:36.390447] I [MSGID: 108006]
     >     [afr-common.c:5677:afr_local_init] 0-vol-data-replicate-0: no
     >     subvolumes up
     >     [2019-10-08 20:10:36.390480] E [MSGID: 101046]
     >     [dht-common.c:1502:dht_lookup_dir_cbk] 0-vol-data-dht: dict
    is null
     >     [2019-10-08 20:10:36.390503] W
     >     [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
     >     00000000-0000-0000-0000-000000000001: failed to resolve
    (Transport
     >     endpoint is not connected)
     >     [2019-10-08 20:10:36.390513] E
    [fuse-bridge.c:928:fuse_getattr_resume]
     >     0-glusterfs-fuse: 3: GETATTR 1
    (00000000-0000-0000-0000-000000000001)
     >     resolution failed
     >     [2019-10-08 20:10:36.390580] I [MSGID: 108006]
     >     [afr-common.c:5677:afr_local_init] 0-vol-data-replicate-0: no
     >     subvolumes up
     >     [2019-10-08 20:10:36.390595] E [MSGID: 101046]
     >     [dht-common.c:1502:dht_lookup_dir_cbk] 0-vol-data-dht: dict
    is null
     >     [2019-10-08 20:10:36.390614] W
     >     [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
     >     00000000-0000-0000-0000-000000000001: failed to resolve
    (Transport
     >     endpoint is not connected)
     >     [2019-10-08 20:10:36.390622] E
    [fuse-bridge.c:928:fuse_getattr_resume]
     >     0-glusterfs-fuse: 4: GETATTR 1
    (00000000-0000-0000-0000-000000000001)
     >     resolution failed
     >     [2019-10-08 20:10:36.410905] I
    [fuse-bridge.c:5199:fuse_thread_proc]
     >     0-fuse: initating unmount of /export/data
     >     [2019-10-08 20:10:36.411091] W
    [glusterfsd.c:1514:cleanup_and_exit]
     >     (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)
    [0x7ff189f586ba]
     >     -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xed) [0x55946f24b70d]
     >     -->/usr/sbin/glusterfs(cleanup_and_exit+0x54)
    [0x55946f24b524] ) 0-:
     >     received signum (15), shutting down
     >     [2019-10-08 20:10:36.411113] I [fuse-bridge.c:5981:fini] 0-fuse:
     >     Unmounting '/export/data'.
     >     [2019-10-08 20:10:36.411122] I [fuse-bridge.c:5986:fini] 0-fuse:
     >     Closing
     >     fuse connection to '/export/data'.
     >     [2019-10-08 20:10:36.845106] I [MSGID: 100030]
    [glusterfsd.c:2741:main]
     >     0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs
    version
     >     4.1.8
     >     (args: /usr/sbin/glusterfs --process-name fuse
     >     --volfile-server=localhost --volfile-id=/vol-data /export/data)
     >     [2019-10-08 20:10:36.848865] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 1
     >     [2019-10-08 20:10:36.852064] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 2
     >     [2019-10-08 20:10:36.852477] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-0: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-08 20:10:36.852694] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-1: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-08 20:10:36.852773] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:36.852877] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:36.852917] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-2: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-08 20:10:36.852947] E [MSGID: 114058]
     >     [client-handshake.c:1442:client_query_portmap_cbk]
    0-vol-data-client-0:
     >     failed to get the port number for remote subvolume. Please
    run 'gluster
     >     volume status' on server to see if brick process is running.
     >     [2019-10-08 20:10:36.852980] I [MSGID: 114018]
     >     [client.c:2254:client_rpc_notify] 0-vol-data-client-0:
    disconnected
     >     from
     >     vol-data-client-0. Client process will keep trying to connect to
     >     glusterd until brick's port is available
     >     [2019-10-08 20:10:36.852990] E [MSGID: 108006]
     >     [afr-common.c:5413:__afr_handle_child_down_event]
     >     0-vol-data-replicate-0: All subvolumes are down. Going
    offline until
     >     atleast one of them comes back up.
     >     [2019-10-08 20:10:37.387355] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:37.387579] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:37.387706] I
    [rpc-clnt.c:2105:rpc_clnt_reconfig]
     >     0-vol-data-client-1: changing port to 49156 (from 0)
     >     [2019-10-08 20:10:37.388065] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:37.388253] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:37.389087] I [MSGID: 114046]
     >     [client-handshake.c:1095:client_setvolume_cbk]
    0-vol-data-client-1:
     >     Connected to vol-data-client-1, attached to remote volume
     >     '/bigdisk/brick2/vol-data'.
     >     [2019-10-08 20:10:37.389102] I [MSGID: 108005]
     >     [afr-common.c:5336:__afr_handle_child_up_event]
    0-vol-data-replicate-0:
     >     Subvolume 'vol-data-client-1' came back up; going online.
     >     [2019-10-08 20:10:39.387062] E
    [socket.c:2524:socket_connect_finish]
     >     0-vol-data-client-2: connection to 134.21.57.122:24007
    <http://134.21.57.122:24007>
     >     <http://134.21.57.122:24007> failed (No route
     >     to host); disconnecting socket
     >     [2019-10-08 20:10:39.389703] I [fuse-bridge.c:4294:fuse_init]
     >     0-glusterfs-fuse: FUSE inited with protocol versions:
    glusterfs 7.24
     >     kernel 7.23
     >     [2019-10-08 20:10:39.389740] I
    [fuse-bridge.c:4927:fuse_graph_sync]
     >     0-fuse: switched to graph 0
     >     [2019-10-08 20:10:39.411859] I
    [glusterfsd-mgmt.c:53:mgmt_cbk_spec]
     >     0-mgmt: Volume file changed
     >     [2019-10-08 20:10:40.832633] E [MSGID: 101046]
     >     [dht-common.c:1502:dht_lookup_dir_cbk] 0-vol-data-dht: dict
    is null
     >     [2019-10-08 20:10:40.832712] E
    [fuse-bridge.c:4362:fuse_first_lookup]
     >     0-fuse: first lookup on root failed (Transport endpoint is not
     >     connected)
     >     [2019-10-08 20:10:40.834248] W
     >     [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
     >     00000000-0000-0000-0000-000000000001: failed to resolve
    (Transport
     >     endpoint is not connected)
     >     [2019-10-08 20:10:40.834281] E
    [fuse-bridge.c:928:fuse_getattr_resume]
     >     0-glusterfs-fuse: 2: GETATTR 1
    (00000000-0000-0000-0000-000000000001)
     >     resolution failed
     >     [2019-10-08 20:10:40.837624] W
     >     [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
     >     00000000-0000-0000-0000-000000000001: failed to resolve
    (Transport
     >     endpoint is not connected)
     >     [2019-10-08 20:10:40.837659] E
    [fuse-bridge.c:928:fuse_getattr_resume]
     >     0-glusterfs-fuse: 3: GETATTR 1
    (00000000-0000-0000-0000-000000000001)
     >     resolution failed
     >     [2019-10-08 20:10:40.839468] W
     >     [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
     >     00000000-0000-0000-0000-000000000001: failed to resolve
    (Transport
     >     endpoint is not connected)
     >     [2019-10-08 20:10:40.839503] E
    [fuse-bridge.c:928:fuse_getattr_resume]
     >     0-glusterfs-fuse: 4: GETATTR 1
    (00000000-0000-0000-0000-000000000001)
     >     resolution failed
     >     [2019-10-08 20:10:40.847013] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:40.847219] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:40.847368] I
    [rpc-clnt.c:2105:rpc_clnt_reconfig]
     >     0-vol-data-client-2: changing port to 49158 (from 0)
     >     [2019-10-08 20:10:40.847725] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:40.847906] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     The message "E [MSGID: 101046]
    [dht-common.c:1502:dht_lookup_dir_cbk]
     >     0-vol-data-dht: dict is null" repeated 3 times between
    [2019-10-08
     >     20:10:40.832633] and [2019-10-08 20:10:40.839454]
     >     [2019-10-08 20:10:40.848759] I [MSGID: 114046]
     >     [client-handshake.c:1095:client_setvolume_cbk]
    0-vol-data-client-2:
     >     Connected to vol-data-client-2, attached to remote volume
     >     '/bigdisk/brick3/vol-data'.
     >     [2019-10-08 20:10:40.848785] I [MSGID: 108002]
     >     [afr-common.c:5611:afr_notify] 0-vol-data-replicate-0:
    Client-quorum
     >     is met
     >     [2019-10-08 20:10:40.874884] I
    [fuse-bridge.c:5199:fuse_thread_proc]
     >     0-fuse: initating unmount of /export/data
     >     [2019-10-08 20:10:40.875054] W
    [glusterfsd.c:1514:cleanup_and_exit]
     >     (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)
    [0x7fdc50b646ba]
     >     -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xed) [0x563108ee670d]
     >     -->/usr/sbin/glusterfs(cleanup_and_exit+0x54)
    [0x563108ee6524] ) 0-:
     >     received signum (15), shutting down
     >     [2019-10-08 20:10:40.875079] I [fuse-bridge.c:5981:fini] 0-fuse:
     >     Unmounting '/export/data'.
     >     [2019-10-08 20:10:40.875087] I [fuse-bridge.c:5986:fini] 0-fuse:
     >     Closing
     >     fuse connection to '/export/data'.
     >     [2019-10-08 20:10:47.464875] I [MSGID: 100030]
    [glusterfsd.c:2741:main]
     >     0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs
    version
     >     4.1.8
     >     (args: /usr/sbin/glusterfs --process-name fuse
     >     --volfile-server=localhost --volfile-id=/vol-data /export/data)
     >     [2019-10-08 20:10:47.468743] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 1
     >     [2019-10-08 20:10:47.472050] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 2
     >     [2019-10-08 20:10:47.472465] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-0: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-08 20:10:47.472803] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-1: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-08 20:10:47.472865] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.472968] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.473036] I
    [rpc-clnt.c:2105:rpc_clnt_reconfig]
     >     0-vol-data-client-0: changing port to 49156 (from 0)
     >     [2019-10-08 20:10:47.473121] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-2: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-08 20:10:47.473466] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.473511] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.473681] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.473850] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.473928] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.474019] I
    [rpc-clnt.c:2105:rpc_clnt_reconfig]
     >     0-vol-data-client-1: changing port to 49156 (from 0)
     >     [2019-10-08 20:10:47.474072] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.474309] I
    [rpc-clnt.c:2105:rpc_clnt_reconfig]
     >     0-vol-data-client-2: changing port to 49158 (from 0)
     >     [2019-10-08 20:10:47.474621] I [MSGID: 114046]
     >     [client-handshake.c:1095:client_setvolume_cbk]
    0-vol-data-client-0:
     >     Connected to vol-data-client-0, attached to remote volume
     >     '/bigdisk/brick1/vol-data'.
     >     [2019-10-08 20:10:47.474638] I [MSGID: 108005]
     >     [afr-common.c:5336:__afr_handle_child_up_event]
    0-vol-data-replicate-0:
     >     Subvolume 'vol-data-client-0' came back up; going online.
     >     [2019-10-08 20:10:47.474750] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.474927] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.474958] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.475216] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-08 20:10:47.476030] I [MSGID: 114046]
     >     [client-handshake.c:1095:client_setvolume_cbk]
    0-vol-data-client-1:
     >     Connected to vol-data-client-1, attached to remote volume
     >     '/bigdisk/brick2/vol-data'.
     >     [2019-10-08 20:10:47.476052] I [MSGID: 108002]
     >     [afr-common.c:5611:afr_notify] 0-vol-data-replicate-0:
    Client-quorum
     >     is met
     >     [2019-10-08 20:10:47.476152] I [MSGID: 114046]
     >     [client-handshake.c:1095:client_setvolume_cbk]
    0-vol-data-client-2:
     >     Connected to vol-data-client-2, attached to remote volume
     >     '/bigdisk/brick3/vol-data'.
     >     [2019-10-08 20:10:47.477159] I [fuse-bridge.c:4294:fuse_init]
     >     0-glusterfs-fuse: FUSE inited with protocol versions:
    glusterfs 7.24
     >     kernel 7.23
     >     [2019-10-08 20:10:47.477210] I
    [fuse-bridge.c:4927:fuse_graph_sync]
     >     0-fuse: switched to graph 0
     >     [2019-10-08 20:10:47.478960] I [MSGID: 108031]
     >     [afr-common.c:2597:afr_local_discovery_cbk]
    0-vol-data-replicate-0:
     >     selecting local read_child vol-data-client-0
     >     [2019-10-08 20:10:47.479971] I [MSGID: 108031]
     >     [afr-common.c:2597:afr_local_discovery_cbk]
    0-vol-data-replicate-0:
     >     selecting local read_child vol-data-client-0
     >     [2019-10-08 20:10:47.480094] I [MSGID: 109005]
     >     [dht-selfheal.c:2342:dht_selfheal_directory] 0-vol-data-dht:
    Directory
     >     selfheal failed: Unable to form layout for directory /
     >     root@nas20:/var/log/glusterfs# cat export-data.log | grep
    "2019-10-09 1"
     >     root@nas20:/var/log/glusterfs# cat export-data.log | grep
    "2019-10-09 7"
     >     root@nas20:/var/log/glusterfs# cat export-data.log | grep
    "2019-10-09 0"
     >     [2019-10-09 04:25:02.165330] I [MSGID: 100011]
     >     [glusterfsd.c:1599:reincarnate] 0-glusterfsd: Fetching the
    volume file
     >     from server...
     >     [2019-10-09 04:25:02.191948] I
     >     [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk]
     >     0-glusterfs: No change in volfile,continuing
     >     [2019-10-09 07:12:03.955619] I [MSGID: 100030]
    [glusterfsd.c:2741:main]
     >     0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs
    version
     >     4.1.8
     >     (args: /usr/sbin/glusterfs --process-name fuse
     >     --volfile-server=localhost --volfile-id=/vol-data /export/data)
     >     [2019-10-09 07:12:03.981652] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 1
     >     [2019-10-09 07:12:04.002485] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 2
     >     [2019-10-09 07:12:04.003899] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-0: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-09 07:12:04.004147] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-1: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-09 07:12:04.004366] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-2: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-09 07:12:04.004628] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:04.004923] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:04.005244] E [MSGID: 114058]
     >     [client-handshake.c:1442:client_query_portmap_cbk]
    0-vol-data-client-0:
     >     failed to get the port number for remote subvolume. Please
    run 'gluster
     >     volume status' on server to see if brick process is running.
     >     [2019-10-09 07:12:04.005286] I [MSGID: 114018]
     >     [client.c:2254:client_rpc_notify] 0-vol-data-client-0:
    disconnected
     >     from
     >     vol-data-client-0. Client process will keep trying to connect to
     >     glusterd until brick's port is available
     >     [2019-10-09 07:12:04.005297] E [MSGID: 108006]
     >     [afr-common.c:5413:__afr_handle_child_down_event]
     >     0-vol-data-replicate-0: All subvolumes are down. Going
    offline until
     >     atleast one of them comes back up.
     >     [2019-10-09 07:12:06.690631] E
    [socket.c:2524:socket_connect_finish]
     >     0-vol-data-client-2: connection to 134.21.57.122:24007
    <http://134.21.57.122:24007>
     >     <http://134.21.57.122:24007> failed (No route
     >     to host); disconnecting socket
     >     [2019-10-09 07:12:06.690792] E
    [socket.c:2524:socket_connect_finish]
     >     0-vol-data-client-1: connection to 192.168.1.121:24007
    <http://192.168.1.121:24007>
     >     <http://192.168.1.121:24007> failed (No route
     >     to host); disconnecting socket
     >     [2019-10-09 07:12:06.691746] I [fuse-bridge.c:4294:fuse_init]
     >     0-glusterfs-fuse: FUSE inited with protocol versions:
    glusterfs 7.24
     >     kernel 7.23
     >     [2019-10-09 07:12:06.691771] I
    [fuse-bridge.c:4927:fuse_graph_sync]
     >     0-fuse: switched to graph 0
     >     The message "E [MSGID: 108006]
     >     [afr-common.c:5413:__afr_handle_child_down_event]
     >     0-vol-data-replicate-0: All subvolumes are down. Going
    offline until
     >     atleast one of them comes back up." repeated 2 times between
     >     [2019-10-09
     >     07:12:04.005297] and [2019-10-09 07:12:06.690811]
     >     [2019-10-09 07:12:06.692647] I [MSGID: 108006]
     >     [afr-common.c:5677:afr_local_init] 0-vol-data-replicate-0: no
     >     subvolumes up
     >     [2019-10-09 07:12:06.692695] E [MSGID: 101046]
     >     [dht-common.c:1502:dht_lookup_dir_cbk] 0-vol-data-dht: dict
    is null
     >     [2019-10-09 07:12:06.692807] E
    [fuse-bridge.c:4362:fuse_first_lookup]
     >     0-fuse: first lookup on root failed (Transport endpoint is not
     >     connected)
     >     [2019-10-09 07:12:06.692955] I [MSGID: 108006]
     >     [afr-common.c:5677:afr_local_init] 0-vol-data-replicate-0: no
     >     subvolumes up
     >     [2019-10-09 07:12:06.692980] E [MSGID: 101046]
     >     [dht-common.c:1502:dht_lookup_dir_cbk] 0-vol-data-dht: dict
    is null
     >     [2019-10-09 07:12:06.693003] W
     >     [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
     >     00000000-0000-0000-0000-000000000001: failed to resolve
    (Transport
     >     endpoint is not connected)
     >     [2019-10-09 07:12:06.693013] E
    [fuse-bridge.c:928:fuse_getattr_resume]
     >     0-glusterfs-fuse: 2: GETATTR 1
    (00000000-0000-0000-0000-000000000001)
     >     resolution failed
     >     [2019-10-09 07:12:06.695503] I [MSGID: 108006]
     >     [afr-common.c:5677:afr_local_init] 0-vol-data-replicate-0: no
     >     subvolumes up
     >     [2019-10-09 07:12:06.695526] E [MSGID: 101046]
     >     [dht-common.c:1502:dht_lookup_dir_cbk] 0-vol-data-dht: dict
    is null
     >     [2019-10-09 07:12:06.695547] W
     >     [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
     >     00000000-0000-0000-0000-000000000001: failed to resolve
    (Transport
     >     endpoint is not connected)
     >     [2019-10-09 07:12:06.695556] E
    [fuse-bridge.c:928:fuse_getattr_resume]
     >     0-glusterfs-fuse: 3: GETATTR 1
    (00000000-0000-0000-0000-000000000001)
     >     resolution failed
     >     [2019-10-09 07:12:06.695619] I [MSGID: 108006]
     >     [afr-common.c:5677:afr_local_init] 0-vol-data-replicate-0: no
     >     subvolumes up
     >     [2019-10-09 07:12:06.695633] E [MSGID: 101046]
     >     [dht-common.c:1502:dht_lookup_dir_cbk] 0-vol-data-dht: dict
    is null
     >     [2019-10-09 07:12:06.695650] W
     >     [fuse-resolve.c:132:fuse_resolve_gfid_cbk] 0-fuse:
     >     00000000-0000-0000-0000-000000000001: failed to resolve
    (Transport
     >     endpoint is not connected)
     >     [2019-10-09 07:12:06.695658] E
    [fuse-bridge.c:928:fuse_getattr_resume]
     >     0-glusterfs-fuse: 4: GETATTR 1
    (00000000-0000-0000-0000-000000000001)
     >     resolution failed
     >     [2019-10-09 07:12:06.714499] I
    [fuse-bridge.c:5199:fuse_thread_proc]
     >     0-fuse: initating unmount of /export/data
     >     [2019-10-09 07:12:06.714753] W
    [glusterfsd.c:1514:cleanup_and_exit]
     >     (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)
    [0x7f133ffef6ba]
     >     -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xed) [0x562b2312c70d]
     >     -->/usr/sbin/glusterfs(cleanup_and_exit+0x54)
    [0x562b2312c524] ) 0-:
     >     received signum (15), shutting down
     >     [2019-10-09 07:12:06.714773] I [fuse-bridge.c:5981:fini] 0-fuse:
     >     Unmounting '/export/data'.
     >     [2019-10-09 07:12:06.714779] I [fuse-bridge.c:5986:fini] 0-fuse:
     >     Closing
     >     fuse connection to '/export/data'.
     >     [2019-10-09 07:12:07.109206] I [MSGID: 100030]
    [glusterfsd.c:2741:main]
     >     0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs
    version
     >     4.1.8
     >     (args: /usr/sbin/glusterfs --process-name fuse
     >     --volfile-server=localhost --volfile-id=/vol-data /export/data)
     >     [2019-10-09 07:12:07.112870] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 1
     >     [2019-10-09 07:12:07.116011] I [MSGID: 101190]
     >     [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll:
    Started thread
     >     with index 2
     >     [2019-10-09 07:12:07.116421] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-0: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-09 07:12:07.116655] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-1: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-09 07:12:07.116676] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:07.116767] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:07.116833] E [MSGID: 114058]
     >     [client-handshake.c:1442:client_query_portmap_cbk]
    0-vol-data-client-0:
     >     failed to get the port number for remote subvolume. Please
    run 'gluster
     >     volume status' on server to see if brick process is running.
     >     [2019-10-09 07:12:07.116835] I [MSGID: 114020]
    [client.c:2328:notify]
     >     0-vol-data-client-2: parent translators are ready, attempting
     >     connect on
     >     transport
     >     [2019-10-09 07:12:07.116887] I [MSGID: 114018]
     >     [client.c:2254:client_rpc_notify] 0-vol-data-client-0:
    disconnected
     >     from
     >     vol-data-client-0. Client process will keep trying to connect to
     >     glusterd until brick's port is available
     >     [2019-10-09 07:12:07.116898] E [MSGID: 108006]
     >     [afr-common.c:5413:__afr_handle_child_down_event]
     >     0-vol-data-replicate-0: All subvolumes are down. Going
    offline until
     >     atleast one of them comes back up.
     >     [2019-10-09 07:12:07.691005] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:09.690613] E
    [socket.c:2524:socket_connect_finish]
     >     0-vol-data-client-2: connection to 134.21.57.122:24007
    <http://134.21.57.122:24007>
     >     <http://134.21.57.122:24007> failed (No route
     >     to host); disconnecting socket
     >     [2019-10-09 07:12:11.111975] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:11.112083] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:11.112200] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:11.112397] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:11.112518] I
    [rpc-clnt.c:2105:rpc_clnt_reconfig]
     >     0-vol-data-client-2: changing port to 49158 (from 0)
     >     [2019-10-09 07:12:11.112820] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:11.113013] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-2: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:09.690664] E [MSGID: 108006]
     >     [afr-common.c:5413:__afr_handle_child_down_event]
     >     0-vol-data-replicate-0: All subvolumes are down. Going
    offline until
     >     atleast one of them comes back up.
     >     [2019-10-09 07:12:11.114003] I [MSGID: 114046]
     >     [client-handshake.c:1095:client_setvolume_cbk]
    0-vol-data-client-2:
     >     Connected to vol-data-client-2, attached to remote volume
     >     '/bigdisk/brick3/vol-data'.
     >     [2019-10-09 07:12:11.114045] I [MSGID: 108005]
     >     [afr-common.c:5336:__afr_handle_child_up_event]
    0-vol-data-replicate-0:
     >     Subvolume 'vol-data-client-2' came back up; going online.
     >     [2019-10-09 07:12:11.290914] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:11.291239] I
    [rpc-clnt.c:2105:rpc_clnt_reconfig]
     >     0-vol-data-client-1: changing port to 49156 (from 0)
     >     [2019-10-09 07:12:11.291676] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:11.291919] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-1: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:11.293266] I [MSGID: 114046]
     >     [client-handshake.c:1095:client_setvolume_cbk]
    0-vol-data-client-1:
     >     Connected to vol-data-client-1, attached to remote volume
     >     '/bigdisk/brick2/vol-data'.
     >     [2019-10-09 07:12:11.293306] I [MSGID: 108002]
     >     [afr-common.c:5611:afr_notify] 0-vol-data-replicate-0:
    Client-quorum
     >     is met
     >     [2019-10-09 07:12:11.295955] I [fuse-bridge.c:4294:fuse_init]
     >     0-glusterfs-fuse: FUSE inited with protocol versions:
    glusterfs 7.24
     >     kernel 7.23
     >     [2019-10-09 07:12:11.296014] I
    [fuse-bridge.c:4927:fuse_graph_sync]
     >     0-fuse: switched to graph 0
     >     [2019-10-09 07:12:11.299181] I [MSGID: 109005]
     >     [dht-selfheal.c:2342:dht_selfheal_directory] 0-vol-data-dht:
    Directory
     >     selfheal failed: Unable to form layout for directory /
     >     [2019-10-09 07:12:14.112691] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:14.112772] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:17.113224] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:17.113319] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:20.113917] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:20.114031] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:24.393064] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:24.393253] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:26.393776] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:26.393880] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:29.394504] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:29.394614] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:32.395375] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:32.395534] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:35.395920] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:35.396027] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:38.396531] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:38.396618] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:41.397419] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:41.397526] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:44.398189] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:44.398312] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:47.399045] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:47.399166] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:50.399735] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:50.399855] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:53.400507] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:53.400616] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:56.401284] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:56.401402] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:59.402080] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:12:59.402200] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:02.402863] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:02.402984] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:05.404125] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:05.404320] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:08.404977] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:08.405172] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:11.405694] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:11.405884] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:14.406443] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:14.406629] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:17.407255] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:17.407445] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:20.408092] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:20.408277] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:23.409546] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:23.409735] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:26.410420] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:26.410600] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:29.411353] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:29.411528] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:32.412325] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:32.412505] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:35.413311] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:35.413491] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:38.414345] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:38.414540] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:41.415407] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:41.415597] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:44.416490] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:44.416672] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:47.417664] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:47.417851] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:50.418814] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:50.419005] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:53.419982] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:53.420166] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:56.421200] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:56.421388] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:59.422450] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:13:59.422630] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:14:02.423757] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:14:02.423952] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:14:05.425051] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:14:05.425243] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:14:08.425832] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:14:08.426011] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:14:11.426636] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:14:11.426846] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:14:16.310279] I
    [glusterfsd-mgmt.c:53:mgmt_cbk_spec]
     >     0-mgmt: Volume file changed
     >     [2019-10-09 07:14:19.393266] I
    [glusterfsd-mgmt.c:53:mgmt_cbk_spec]
     >     0-mgmt: Volume file changed
     >     [2019-10-09 07:14:19.465709] I
     >     [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk]
     >     0-glusterfs: No change in volfile,continuing
     >     [2019-10-09 07:14:19.467466] I
     >     [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk]
     >     0-glusterfs: No change in volfile,continuing
     >     [2019-10-09 07:14:29.457122] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:14:29.457312] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:14:29.457431] I
    [rpc-clnt.c:2105:rpc_clnt_reconfig]
     >     0-vol-data-client-0: changing port to 49157 (from 0)
     >     [2019-10-09 07:14:29.458078] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:14:29.458264] W [rpc-clnt.c:1753:rpc_clnt_submit]
     >     0-vol-data-client-0: error returned while attempting to
    connect to
     >     host:(null), port:0
     >     [2019-10-09 07:14:29.459212] I [MSGID: 114046]
     >     [client-handshake.c:1095:client_setvolume_cbk]
    0-vol-data-client-0:
     >     Connected to vol-data-client-0, attached to remote volume
     >     '/bigdisk/brick1/vol-data'.
     >
     >     Regards,
     >     Birgit
     >
     >     On 13/10/19 08:13, Amar Tumballi wrote:
     >      > 'Transport endpoint not connected' (ie, ENOTCONN) comes
    when the n/w
     >      > connection is not established between client and the server. I
     >     recommend
     >      > checking the logs for particular reason. Specially the
    brick (server
     >      > side) logs will have some hints on this.
     >      >
     >      > About the crash, we treat it as a bug. Considering there is no
     >     specific
     >      > backtrace, or logs shared with the email, it is hard to
    tell if
     >     it is
     >      > already fixed in higher version or not.
     >      >
     >      > Considering you are in 4.1.8 version, and there are many
    releases
     >     done
     >      > after that, upgrading also can be an option.
     >      >
     >      > Regards,
     >      > Amar
     >      >
     >      >
     >      > On Fri, Oct 11, 2019 at 4:13 PM DUCARROZ Birgit
     >      > <birgit.ducarroz@xxxxxxxx
    <mailto:birgit.ducarroz@xxxxxxxx> <mailto:birgit.ducarroz@xxxxxxxx
    <mailto:birgit.ducarroz@xxxxxxxx>>
     >     <mailto:birgit.ducarroz@xxxxxxxx
    <mailto:birgit.ducarroz@xxxxxxxx> <mailto:birgit.ducarroz@xxxxxxxx
    <mailto:birgit.ducarroz@xxxxxxxx>>>>
     >     wrote:
     >      >
     >      >     Hi list,
     >      >
     >      >     Does anyone know what I can do to avoid "Transport
    Endpoint not
     >      >     connected" (and then to get a blocked server) when
    writing a
     >     lot of
     >      >     small files on a volume?
     >      >
     >      >     I'm running glusterfs 4.1.8 on 6 servers. With 3 servers I
     >     never have
     >      >     problems, but the other 3 servers are acting as HA storage
     >     for people
     >      >     who write sometimes a thousands of small files. This
    seems to
     >     provoke a
     >      >     crash of the gluster daemon.
     >      >
     >      >     I have 3 bricks whereas the 3rd brick acts as arbiter.
     >      >
     >      >
     >      >     # Location of the bricks:
     >      >     #-------$HOST1-------  -------$HOST3-------
     >      >     # brick1            |  | brick3           | brick3 =
    arbiter
     >      >     #                   |  |                  |
     >      >     #-------$HOST2-------  --------------------
     >      >     # brick2            |
     >      >     #--------------------
     >      >
     >      >     Checked:
     >      >     The underlying ext4 filesystem and the HD's seem to be
     >     without errors.
     >      >     The ports in the firewall should not be the problem
    since it
     >     occurs
     >      >     also
     >      >     when the firewall is disabled.
     >      >
     >      >     Any help appreciated!
     >      >     Kind regards,
     >      >     Birgit
     >      >     ________
     >      >
     >      >     Community Meeting Calendar:
     >      >
     >      >     APAC Schedule -
     >      >     Every 2nd and 4th Tuesday at 11:30 AM IST
     >      >     Bridge: https://bluejeans.com/118564314
     >      >
     >      >     NA/EMEA Schedule -
     >      >     Every 1st and 3rd Tuesday at 01:00 PM EDT
     >      >     Bridge: https://bluejeans.com/118564314
     >      >
     >      >     Gluster-users mailing list
     >      > Gluster-users@xxxxxxxxxxx
    <mailto:Gluster-users@xxxxxxxxxxx> <mailto:Gluster-users@xxxxxxxxxxx
    <mailto:Gluster-users@xxxxxxxxxxx>>
     >     <mailto:Gluster-users@xxxxxxxxxxx
    <mailto:Gluster-users@xxxxxxxxxxx> <mailto:Gluster-users@xxxxxxxxxxx
    <mailto:Gluster-users@xxxxxxxxxxx>>>
     >      > https://lists.gluster.org/mailman/listinfo/gluster-users
     >      >
     >

________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/118564314

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/118564314

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users




[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux