Ok, I setup a four brick AFR test and it is working. Well at least as
long as nothing goes wrong. But if I take down one of the servers
(glusterfsd) and try to write to any of the mounts then the whole thing
hangs. This is the type of situation that I am hoping will be handled
gracefully. I've had tcp/ip stacks hang, servers hang and need rebooted
and once back up the whole cluster needs to be able to recover itself.
So with the current state of GlusterFS is this a bug or is there
functionality planned that would support this level of robustness.
[root@grp-01-30-01 glusterfs]# touch /mnt/glusterfs3/file3
[root@grp-01-30-01 glusterfs]# find /mnt/glusterfs?
/mnt/glusterfs0
/mnt/glusterfs0/file0
/mnt/glusterfs0/file3
/mnt/glusterfs0/file2
/mnt/glusterfs0/file1
/mnt/glusterfs1
/mnt/glusterfs1/file0
/mnt/glusterfs1/file3
/mnt/glusterfs1/file2
/mnt/glusterfs1/file1
/mnt/glusterfs2
/mnt/glusterfs2/file0
/mnt/glusterfs2/file3
/mnt/glusterfs2/file2
/mnt/glusterfs2/file1
/mnt/glusterfs3
/mnt/glusterfs3/file0
/mnt/glusterfs3/file3
/mnt/glusterfs3/file2
/mnt/glusterfs3/file1
[root@grp-01-30-01 glusterfs]# find /root/export? #underlying filesystems
/root/export0
/root/export0/file1
/root/export0/file2
/root/export0/file3
/root/export0/file0
/root/export1
/root/export1/file1
/root/export1/file2
/root/export1/file3
/root/export1/file0
/root/export2
/root/export2/file1
/root/export2/file2
/root/export2/file3
/root/export2/file0
/root/export3
/root/export3/file1
/root/export3/file2
/root/export3/file3
/root/export3/file0
=================================
now kill one server:
[root@grp-01-30-01 glusterfs]# kill -9 23879
[root@grp-01-30-01 glusterfs]# ps -e | grep gluster
23881 ? 00:00:00 glusterfsd
23883 ? 00:00:00 glusterfsd
23885 ? 00:00:00 glusterfsd
23888 ? 00:00:00 glusterfs
23891 ? 00:00:00 glusterfs
23894 ? 00:00:00 glusterfs
23897 ? 00:00:00 glusterfs
[root@grp-01-30-01 glusterfs]# find /mnt/glusterfs?
/mnt/glusterfs0
/mnt/glusterfs0/file0
/mnt/glusterfs0/file2
/mnt/glusterfs0/file1
/mnt/glusterfs1
/mnt/glusterfs1/file0
/mnt/glusterfs1/file2
/mnt/glusterfs1/file1
/mnt/glusterfs2
/mnt/glusterfs2/file0
/mnt/glusterfs2/file2
/mnt/glusterfs2/file1
/mnt/glusterfs3
/mnt/glusterfs3/file0
/mnt/glusterfs3/file2
/mnt/glusterfs3/file1
=== now touch a new file =====
[root@grp-01-30-01 glusterfs]# touch /mnt/glusterfs1/file-0down
=== ^C done here
[root@grp-01-30-01 glusterfs]# find /mnt/glusterfs?
/mnt/glusterfs0
/mnt/glusterfs0/file0
/mnt/glusterfs0/file2
/mnt/glusterfs0/file1
=== ^C done here
Both commands hung.
Gerry
LOG OUTPUT:
[Apr 04 20:55:24] [ERROR/common-utils.c:54/full_rw()]
libglusterfs:full_rw: 0 bytes r/w instead of 113
[Apr 04 20:55:24]
[DEBUG/protocol.c:244/gf_block_unserialize_transport()]
libglusterfs/protocol:gf_block_unserialize_transport: full_read of
header failed
[Apr 04 20:55:24]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:24] [ERROR/common-utils.c:54/full_rw()]
libglusterfs:full_rw: 0 bytes r/w instead of 113
[Apr 04 20:55:24]
[DEBUG/protocol.c:244/gf_block_unserialize_transport()]
libglusterfs/protocol:gf_block_unserialize_transport: full_read of
header failed
[Apr 04 20:55:24]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:24] [ERROR/common-utils.c:54/full_rw()]
libglusterfs:full_rw: 0 bytes r/w instead of 113
[Apr 04 20:55:24]
[DEBUG/protocol.c:244/gf_block_unserialize_transport()]
libglusterfs/protocol:gf_block_unserialize_transport: full_read of
header failed
[Apr 04 20:55:24]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:24] [ERROR/common-utils.c:54/full_rw()]
libglusterfs:full_rw: 0 bytes r/w instead of 113
[Apr 04 20:55:24]
[DEBUG/protocol.c:244/gf_block_unserialize_transport()]
libglusterfs/protocol:gf_block_unserialize_transport: full_read of
header failed
[Apr 04 20:55:24]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:56:13] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:56:13] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:56:13] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:13] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:13] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:13]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:56:13] [DEBUG/afr.c:65/afr_get_num_copies()] afr:matched!
pattern = *, filename = file-0down,
[Apr 04 20:56:13] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:56:13] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:56:13] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:13] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:13] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:13]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()]
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()]
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()]
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37]
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()]
protocol/client:cleaning up state in transport object 0x885fa08
