Hi Gerry, We were trying to see how glusterfs behaves when "cp -ax" is started and while cp is happening, kill all the servers. The log messages which were produced were same as the ones you have given, So are you sure that glusterfsd processes were not dead when cp was happening? ( Connection refused would indicate that glusterfsd is not listening at that port) We are checking how the log messages can be avoided when this particular situation is encountered. Thanks Krishna On 7/4/07, Gerry Reno <greno@xxxxxxxxxxx> wrote:
Actions: [root@grp-01-20-01 mnt]# cd glusterfs2 [root@grp-01-20-01 glusterfs2]# ls dir1 file2 file3 file333 local [root@grp-01-20-01 glusterfs2]# rm -rf local [root@grp-01-20-01 glusterfs2]# [root@grp-01-20-01 glusterfs2]# [root@grp-01-20-01 glusterfs2]# [root@grp-01-20-01 glusterfs2]# find /mnt/glusterfs* /mnt/glusterfs0 /mnt/glusterfs0/dir1 /mnt/glusterfs0/dir1/file0 /mnt/glusterfs0/file3 /mnt/glusterfs0/file333 /mnt/glusterfs0/file2 /mnt/glusterfs1 /mnt/glusterfs1/dir1 /mnt/glusterfs1/dir1/file0 /mnt/glusterfs1/file3 /mnt/glusterfs1/file333 /mnt/glusterfs1/file2 /mnt/glusterfs2 /mnt/glusterfs2/dir1 /mnt/glusterfs2/dir1/file0 /mnt/glusterfs2/file3 /mnt/glusterfs2/file333 /mnt/glusterfs2/file2 /mnt/glusterfs3 /mnt/glusterfs3/dir1 /mnt/glusterfs3/dir1/file0 /mnt/glusterfs3/file3 /mnt/glusterfs3/file333 /mnt/glusterfs3/file2 [root@grp-01-20-01 glusterfs2]# cd .. [root@grp-01-20-01 mnt]# cp -ax /usr /mnt/glusterfs0 cp: cannot create directory `/mnt/glusterfs0/usr': Device or resource busy [root@grp-01-20-01 mnt]# In the log I see this: 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3: non-blocking connect() returned: 111 (Connection refused) 2007-07-04 09:24:55 E [client-protocol.c:3780:client_lookup_cbk] client3: client lookup failed 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3: non-blocking connect() returned: 111 (Connection refused) 2007-07-04 09:24:55 E [client-protocol.c:3780:client_lookup_cbk] client3: client lookup failed 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3: non-blocking connect() returned: 111 (Connection refused) 2007-07-04 09:24:55 E [client-protocol.c:3780:client_lookup_cbk] client3: client lookup failed 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3: non-blocking connect() returned: 111 (Connection refused) 2007-07-04 09:24:55 E [client-protocol.c:3780:client_lookup_cbk] client3: client lookup failed 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3: non-blocking connect() returned: 111 (Connection refused) 2007-07-04 09:24:55 E [tcp-client.c:171:tcp_connect] client3: non-blocking connect() returned: 111 (Connection refused) repeated tens of thousands of times The log is growing rapidly. It is now 3G. Log attached. About ten minutes later I retried the same command without doing anything to the servers or clients and it is running: [root@grp-01-20-01 mnt]# cp -ax /usr /mnt/glusterfs0 This behavior is very strange. Gerry
