Hi Ravi,
back to our client-cannot-reconnect-to-gluster-brick problem ...
> Von: Ravishankar N [ravishankar@xxxxxxxxxx]
> Gesendet: Montag, 29. Mai 2017 06:34
> An: Markus Stockhausen; gluster-users@xxxxxxxxxxx
> Betreff: Re: gluster heal entry reappears
>
> > On 05/28/2017 10:31 PM, Markus Stockhausen wrote:
> > Hi,
> >
> > I'm fairly new to gluster and quite happy with it. We are using it in an OVirt
> > environment that stores its VM images in the gluster. Setup is as follows and
> > Clients mount the volume with gluster native fuse protocol.
> >
> > 3 storage nodes: Centos 7, Gluster 3.8.12 (managed by me), 2 bricks each
> > 5 virtualization nodes: Centos 7, Gluster 3.8.12 (managed by OVirt engine)
> >
> > After todays reboot of one of the storage nodes the recovery did not finish
> > successfully. The state of one brick remained in:
> >
> > [root@cfiler301 dom_md]# gluster volume heal gluster1 info
> > ...
> > Brick cfilers201:/var/data/brick1/brick
> > /b1de7818-020b-4f47-938f-f3ebb51836a3/dom_md/ids
> > Status: Connected
> > Number of entries: 1
> > ...
> >
> > The above file is used by sanlock runing on the OVirt nodes to handle VM
> > image locking. Issuing a manual heal with "gluster volume heal gluster1" fixed
> > the problem but the unsynced entry reappeared a few seconds later.
> >
> > My question: Should this situation be recovered automatically and if yes
> > what might be the culprit?
>
> We have had this observed by our QE guys while testing too, but in all cases, there was an intermittent disconnect from the fuse mount to the bricks, leading to the 'ids' file needing heal (and being healed on reconnect) again and again.
> Perhaps you should check if and why the mount is getting disconnected from the bricks.
>
> HTH,
> Ravi
Just finished a maintenance of one cluster brick node
and the error reappeared. Status is as follows:
- 3 gluster brick nodes: working flawlessley. logs are silent after resync
- 3 gluster fuse client nodes: working flawlessley. logs are silent after resync
- 1 gluster fuse client node: spitting errors every 2 minutes since the maintenane
Client log beginning at the maintenance time:
[2017-06-17 09:08:26.032957] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gluster1-client-1: server 100.64.252.1:49154 has not responded in the last 42 seconds, disconnecting.
[2017-06-17 09:08:26.033302] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gluster1-client-2: server 100.64.252.1:49155 has not responded in the last 42 seconds, disconnecting.
[2017-06-17 09:08:26.033751] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] )))))))))) 0-gluster1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-06-17 09:07:43.578036 (xid=0x10e233)
[2017-06-17 09:08:26.033753] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] )))))))))) 0-gluster1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-06-17 09:07:59.914614 (xid=0xa4141)
[2017-06-17 09:08:26.033788] E [MSGID: 114031] [client-rpc-fops.c:1596:client3_3_finodelk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]
[2017-06-17 09:08:26.033799] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gluster1-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2017-06-17 09:08:26.034008] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-2: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-06-17 09:07:43.578069 (xid=0xa413e)
[2017-06-17 09:08:26.034029] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gluster1-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2017-06-17 09:08:26.034078] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-1: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2017-06-17 09:07:44.028106 (xid=0x10e234)
[2017-06-17 09:08:26.034150] W [rpc-clnt-ping.c:203:rpc_clnt_ping_cbk] 0-gluster1-client-1: socket disconnected
[2017-06-17 09:08:26.034345] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-06-17 09:07:48.933278 (xid=0x10e235)
[2017-06-17 09:08:26.034459] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-2: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2017-06-17 09:07:44.028115 (xid=0xa413f)
[2017-06-17 09:08:26.034478] W [rpc-clnt-ping.c:203:rpc_clnt_ping_cbk] 0-gluster1-client-2: socket disconnected
[2017-06-17 09:08:26.034535] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-06-17 09:08:01.714032 (xid=0x10e236)
[2017-06-17 09:08:26.034614] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-2: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-06-17 09:07:48.933304 (xid=0xa4140)
[2017-06-17 09:08:26.034820] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-2: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-06-17 09:08:01.714058 (xid=0xa4142)
[2017-06-17 09:08:26.034923] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-gluster1-client-2: disconnected from gluster1-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2017-06-17 09:08:26.037127] I [socket.c:3401:socket_submit_request] 0-gluster1-client-1: not connected (priv->connected = 0)
[2017-06-17 09:08:26.037145] W [rpc-clnt.c:1657:rpc_clnt_submit] 0-gluster1-client-1: failed to submit rpc-request (XID: 0x10e237 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gluster1-client-1)
[2017-06-17 09:08:26.037172] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gluster1-client-1: remote operation failed. Path: /b1de7818-020b-4f47-938f-f3ebb51836a3 (663eae40-1818-43bb-b479-dec81057b5e1) [Transport endpoint is not connected]
[2017-06-17 09:08:26.037195] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-gluster1-client-1: disconnected from gluster1-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2017-06-17 09:08:26.038702] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gluster1-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2017-06-17 09:08:26.121603] W [MSGID: 114031] [client-rpc-fops.c:2524:client3_3_lk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]
[2017-06-17 09:08:29.040058] E [socket.c:2309:socket_connect_finish] 0-gluster1-client-1: connection to 100.64.252.1:24007 failed (No route to host)
[2017-06-17 09:08:38.142061] E [socket.c:2309:socket_connect_finish] 0-gluster1-client-2: connection to 100.64.252.1:24007 failed (No route to host)
The message "W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gluster1-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]" repeated 2 times between [2017-06-17 09:08:26.033799] and [2017-06-17 09:08:26.034553]
The message "W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gluster1-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]" repeated 2 times between [2017-06-17 09:08:26.034029] and [2017-06-17 09:08:26.034850]
[2017-06-17 09:08:26.035008] E [MSGID: 114031] [client-rpc-fops.c:1596:client3_3_finodelk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114031] [client-rpc-fops.c:2524:client3_3_lk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2017-06-17 09:08:26.121603] and [2017-06-17 09:09:01.336388]
[2017-06-17 09:10:01.462178] W [MSGID: 114031] [client-rpc-fops.c:2524:client3_3_lk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114031] [client-rpc-fops.c:2524:client3_3_lk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2017-06-17 09:10:01.462178] and [2017-06-17 09:11:01.668207]
[2017-06-17 09:12:01.792341] W [MSGID: 114031] [client-rpc-fops.c:2524:client3_3_lk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114031] [client-rpc-fops.c:2524:client3_3_lk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2017-06-17 09:12:01.792341] and [2017-06-17 09:13:02.002464]
...
And so on
...
Digging deeper into the problems with an strace of the gluster process
on the client I found the following sequence (100.64.252.1 being the
brick that was in maintenance):
4543 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 5
4543 setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
4543 setsockopt(5, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
4543 setsockopt(5, SOL_TCP, TCP_KEEPIDLE, [20], 4) = 0
4543 setsockopt(5, SOL_TCP, TCP_KEEPINTVL, [2], 4) = 0
4543 setsockopt(5, SOL_TCP, TCP_USER_TIMEOUT, [0], 4) = 0
4543 open("/proc/sys/net/ipv4/ip_local_reserved_ports", O_RDONLY) = 7
4543 read(7, "\n", 4096) = 1
4543 close(7) = 0
4543 bind(5, {sa_family=AF_INET, sin_port=htons(49151), sin_addr=inet_addr("100.64.251.6")}, 16) = 0
4543 fcntl(5, F_GETFL) = 0x2 (flags O_RDWR)
4543 fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
4543 connect(5, {sa_family=AF_INET, sin_port=htons(24007), sin_addr=inet_addr("100.64.252.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
4543 fcntl(5, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
4543 fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
4543 epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLPRI|EPOLLOUT|EPOLLONESHOT, {u32=2, u64=4952097292290}}) = 0
...
4543 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 7
4543 setsockopt(7, SOL_TCP, TCP_NODELAY, [1], 4) = 0
4543 setsockopt(7, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
4543 setsockopt(7, SOL_TCP, TCP_KEEPIDLE, [20], 4) = 0
4543 setsockopt(7, SOL_TCP, TCP_KEEPINTVL, [2], 4) = 0
4543 setsockopt(7, SOL_TCP, TCP_USER_TIMEOUT, [0], 4) = 0
4543 open("/proc/sys/net/ipv4/ip_local_reserved_ports", O_RDONLY) = 12
4543 read(12, "\n", 4096) = 1
4543 close(12) = 0
4543 bind(7, {sa_family=AF_INET, sin_port=htons(49151), sin_addr=inet_addr("100.64.251.6")}, 16) = -1 EADDRINUSE (Address already in use)
4543 bind(7, {sa_family=AF_INET, sin_port=htons(49150), sin_addr=inet_addr("100.64.251.6")}, 16) = -1 EADDRINUSE (Address already in use)
4543 bind(7, {sa_family=AF_INET, sin_port=htons(49149), sin_addr=inet_addr("100.64.251.6")}, 16) = 0
4543 fcntl(7, F_GETFL) = 0x2 (flags O_RDWR)
4543 fcntl(7, F_SETFL, O_RDWR|O_NONBLOCK) = 0
4543 connect(7, {sa_family=AF_INET, sin_port=htons(24007), sin_addr=inet_addr("100.64.252.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
4543 fcntl(7, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
4543 fcntl(7, F_SETFL, O_RDWR|O_NONBLOCK) = 0
4543 epoll_ctl(3, EPOLL_CTL_ADD, 7, {EPOLLIN|EPOLLPRI|EPOLLOUT|EPOLLONESHOT, {u32=4, u64=4810363371524}}) = 0
...
4550 <... epoll_wait resumed> {{EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=2, u64=4952097292290}}}, 1, -1) = 1
4548 <... epoll_wait resumed> {{EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=4, u64=4810363371524}}}, 1, -1) = 1
4550 getsockopt(5, SOL_SOCKET, SO_ERROR, <unfinished ...>
4548 getsockopt(7, SOL_SOCKET, SO_ERROR, <unfinished ...>
4550 <... getsockopt resumed> [110], [4]) = 0
4548 <... getsockopt resumed> [110], [4]) = 0
4550 shutdown(5, SHUT_RDWR <unfinished ...>
4548 shutdown(7, SHUT_RDWR <unfinished ...>
4550 <... shutdown resumed> ) = -1 ENOTCONN (Transport endpoint is not connected)
4548 <... shutdown resumed> ) = -1 ENOTCONN (Transport endpoint is not connected)
...
Direct ping+ssh between server and client (and vice versa) works fine.
Looking at a tcpdump on the brick side I see the TCP handshake for the
client. Looks strange because it is repeated after 1,2,4 seconds and so
on
18:55:51.176832 IP cadmins101.49151 > cfilers201.24007: Flags [S], seq 3453818375, win 28364, options [mss 4052,sackOK,TS val 362061385 ecr 0,nop,wscale 7], length 0
18:55:51.176884 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 1929363096, ack 3453818376, win 28280, options [mss 4052,sackOK,TS val 13791464 ecr 362061385,nop,wscale 7], length 0
18:55:51.179391 IP cadmins101.49149 > cfilers201.24007: Flags [S], seq 2219902567, win 28364, options [mss 4052,sackOK,TS val 362061388 ecr 0,nop,wscale 7], length 0
18:55:51.179433 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 2413229577, ack 2219902568, win 28280, options [mss 4052,sackOK,TS val 13791466 ecr 362061388,nop,wscale 7], length 0
18:55:52.176704 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 1929363096, ack 3453818376, win 28280, options [mss 4052,sackOK,TS val 13792464 ecr 362061385,nop,wscale 7], length 0
18:55:52.178695 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 2413229577, ack 2219902568, win 28280, options [mss 4052,sackOK,TS val 13792466 ecr 362061388,nop,wscale 7], length 0
18:55:52.178951 IP cadmins101.49151 > cfilers201.24007: Flags [S], seq 3453818375, win 28364, options [mss 4052,sackOK,TS val 362062388 ecr 0,nop,wscale 7], length 0
18:55:52.178971 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 1929363096, ack 3453818376, win 28280, options [mss 4052,sackOK,TS val 13792466 ecr 362061385,nop,wscale 7], length 0
18:55:52.180951 IP cadmins101.49149 > cfilers201.24007: Flags [S], seq 2219902567, win 28364, options [mss 4052,sackOK,TS val 362062390 ecr 0,nop,wscale 7], length 0
18:55:52.180979 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 2413229577, ack 2219902568, win 28280, options [mss 4052,sackOK,TS val 13792468 ecr 362061388,nop,wscale 7], length 0
18:55:54.178704 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 1929363096, ack 3453818376, win 28280, options [mss 4052,sackOK,TS val 13794466 ecr 362061385,nop,wscale 7], length 0
18:55:54.180704 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 2413229577, ack 2219902568, win 28280, options [mss 4052,sackOK,TS val 13794468 ecr 362061388,nop,wscale 7], length 0
18:55:54.182952 IP cadmins101.49151 > cfilers201.24007: Flags [S], seq 3453818375, win 28364, options [mss 4052,sackOK,TS val 362064392 ecr 0,nop,wscale 7], length 0
18:55:54.182982 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 1929363096, ack 3453818376, win 28280, options [mss 4052,sackOK,TS val 13794470 ecr 362061385,nop,wscale 7], length 0
18:55:54.182991 IP cadmins101.49149 > cfilers201.24007: Flags [S], seq 2219902567, win 28364, options [mss 4052,sackOK,TS val 362064392 ecr 0,nop,wscale 7], length 0
18:55:54.183004 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 2413229577, ack 2219902568, win 28280, options [mss 4052,sackOK,TS val 13794470 ecr 362061388,nop,wscale 7], length 0
Same on the client side (tcpdump at later time):
19:02:18.241460 IP cadmins101.49151 > cfilers201.24007: Flags [S], seq 911752495, win 28364, options [mss 4052,sackOK,TS val 362448449 ecr 0,nop,wscale 7], length 0
19:02:18.241642 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 3682274790, ack 911752496, win 28280, options [mss 4052,sackOK,TS val 14178528 ecr 362448449,nop,wscale 7], length 0
19:02:18.243955 IP cadmins101.49149 > cfilers201.24007: Flags [S], seq 3972802780, win 28364, options [mss 4052,sackOK,TS val 362448452 ecr 0,nop,wscale 7], length 0
19:02:18.244132 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 4166140203, ack 3972802781, win 28280, options [mss 4052,sackOK,TS val 14178530 ecr 362448452,nop,wscale 7], length 0
19:02:19.241228 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 3682274790, ack 911752496, win 28280, options [mss 4052,sackOK,TS val 14179528 ecr 362448449,nop,wscale 7], length 0
19:02:19.243220 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 4166140203, ack 3972802781, win 28280, options [mss 4052,sackOK,TS val 14179530 ecr 362448452,nop,wscale 7], length 0
19:02:19.244000 IP cadmins101.49151 > cfilers201.24007: Flags [S], seq 911752495, win 28364, options [mss 4052,sackOK,TS val 362449452 ecr 0,nop,wscale 7], length 0
19:02:19.244121 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 3682274790, ack 911752496, win 28280, options [mss 4052,sackOK,TS val 14179530 ecr 362448449,nop,wscale 7], length 0
19:02:19.245995 IP cadmins101.49149 > cfilers201.24007: Flags [S], seq 3972802780, win 28364, options [mss 4052,sackOK,TS val 362449454 ecr 0,nop,wscale 7], length 0
19:02:19.246097 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 4166140203, ack 3972802781, win 28280, options [mss 4052,sackOK,TS val 14179532 ecr 362448452,nop,wscale 7], length 0
19:02:21.243229 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 3682274790, ack 911752496, win 28280, options [mss 4052,sackOK,TS val 14181530 ecr 362448449,nop,wscale 7], length 0
19:02:21.245222 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 4166140203, ack 3972802781, win 28280, options [mss 4052,sackOK,TS val 14181532 ecr 362448452,nop,wscale 7], length 0
19:02:21.247997 IP cadmins101.49151 > cfilers201.24007: Flags [S], seq 911752495, win 28364, options [mss 4052,sackOK,TS val 362451456 ecr 0,nop,wscale 7], length 0
19:02:21.248007 IP cadmins101.49149 > cfilers201.24007: Flags [S], seq 3972802780, win 28364, options [mss 4052,sackOK,TS val 362451456 ecr 0,nop,wscale 7], length 0
19:02:21.248130 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 3682274790, ack 911752496, win 28280, options [mss 4052,sackOK,TS val 14181534 ecr 362448449,nop,wscale 7], length 0
19:02:21.248173 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 4166140203, ack 3972802781, win 28280, options [mss 4052,sackOK,TS val 14181534 ecr 362448452,nop,wscale 7], length 0
Out of ideas now. Maybe you see something.
Markus****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.
�ber das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
Vorstand:
Kadir Akin
Dr. Michael Höhnerbach
Vorsitzender des Aufsichtsrates:
Hans Kristian Langva
Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
executive board:
Kadir Akin
Dr. Michael Höhnerbach
President of the supervisory board:
Hans Kristian Langva
Registry office: district court Cologne
Register number: HRB 52 497
****************************************************************************
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users
