Looks like vs0 crashed first and vs1 and vs2 about the same time after that. I am running fuse-2.7.0-glfs1 and glusterfs--mainline--2.5--patch-410 http://share.robotics.net/vs0_server.vol http://share.robotics.net/vs0_glusterfsd.log http://share.robotics.net/vs1_server.vol http://share.robotics.net/vs1_glusterfsd.log http://share.robotics.net/vs2_server.vol http://share.robotics.net/vs2_glusterfsd.log (vs0) 2007-08-02 13:46:22 E [tcp.c:118:tcp_except] server: shutdown () - error: Transport endpoint is not connected --------- got signal (11), printing backtrace --------- /lib64/libc.so.6[0x3f604300c0] /usr/local/lib/glusterfs/1.3.0/xlator/protocol/server.so[0x2aaaabb328a4] /usr/local/lib/glusterfs/1.3.0/xlator/protocol/server.so[0x2aaaabb32d63] /usr/local/lib/glusterfs/1.3.0/xlator/performance/io-threads.so[0x2aaaab92bfd5] /usr/local/lib/glusterfs/1.3.0/xlator/cluster/unify.so(unify_writev_cbk+0x19)[0x2aaaab71e3b9] /usr/local/lib/glusterfs/1.3.0/xlator/cluster/afr.so[0x2aaaab50de59] /usr/local/lib/glusterfs/1.3.0/xlator/protocol/client.so[0x2aaaab305a19] /usr/local/lib/glusterfs/1.3.0/xlator/cluster/afr.so[0x2aaaab5100c8] /usr/local/lib/glusterfs/1.3.0/xlator/cluster/unify.so(unify_writev+0xf7)[0x2aaaab722097] /usr/local/lib/glusterfs/1.3.0/xlator/performance/io-threads.so[0x2aaaab92c873] /usr/local/lib/libglusterfs.so.0[0x2aaaaaad7df4] /usr/local/lib/libglusterfs.so.0(call_resume+0x6a)[0x2aaaaaad826a] /usr/local/lib/glusterfs/1.3.0/xlator/performance/io-threads.so[0x2aaaab92dded] /lib64/libpthread.so.0[0x3f610062f7] /lib64/libc.so.6(clone+0x6d)[0x3f604ce86d] --------- (vs1) 2007-08-02 13:46:25 D [tcp-client.c:142:tcp_connect] brick-a-ns: connect on 8 in progress (non-blocking) --------- got signal (11), printing backtrace --------- /lib64/libc.so.6[0x3f604300c0] /usr/local/lib/glusterfs/1.3.0/xlator/protocol/server.so[0x2aaaabb328a4] /usr/local/lib/glusterfs/1.3.0/xlator/protocol/server.so[0x2aaaabb32d63] /usr/local/lib/glusterfs/1.3.0/xlator/performance/io-threads.so[0x2aaaab92bfd5] /usr/local/lib/glusterfs/1.3.0/xlator/cluster/unify.so(unify_writev_cbk+0x19)[0x2aaaab71e3b9] /usr/local/lib/glusterfs/1.3.0/xlator/cluster/afr.so[0x2aaaab50de59] /usr/local/lib/glusterfs/1.3.0/xlator/protocol/client.so[0x2aaaab305a19] /usr/local/lib/glusterfs/1.3.0/xlator/cluster/afr.so[0x2aaaab5100c8] /usr/local/lib/glusterfs/1.3.0/xlator/cluster/unify.so(unify_writev+0xf7)[0x2aaaab722097] /usr/local/lib/glusterfs/1.3.0/xlator/performance/io-threads.so[0x2aaaab92c873] /usr/local/lib/libglusterfs.so.0[0x2aaaaaad7df4] /usr/local/lib/libglusterfs.so.0(call_resume+0x6a)[0x2aaaaaad826a] /usr/local/lib/glusterfs/1.3.0/xlator/performance/io-threads.so[0x2aaaab92dded] /lib64/libpthread.so.0[0x3f610062f7] /lib64/libc.so.6(clone+0x6d)[0x3f604ce86d] --------- (vs2) 2007-08-02 13:46:25 E [afr.c:1757:afr_readv_cbk] block-b-ds-afr: (path=/vs0_video1_13_2007-08-02.rs child=brick-b-ds) op_ret=-1 op_errno=107 /lib64/libc.so.6[0x3f604300c0] /usr/local/lib/glusterfs/1.3.0/xlator/protocol/server.so[0x2aaaabb328a4] /usr/local/lib/glusterfs/1.3.0/xlator/protocol/server.so[0x2aaaabb32d63] /usr/local/lib/glusterfs/1.3.0/xlator/performance/io-threads.so[0x2aaaab92bfd5] /usr/local/lib/glusterfs/1.3.0/xlator/cluster/unify.so(unify_writev_cbk+0x19)[0x2aaaab71e3b9] /usr/local/lib/glusterfs/1.3.0/xlator/cluster/afr.so[0x2aaaab50de59] /usr/local/lib/glusterfs/1.3.0/xlator/protocol/client.so[0x2aaaab305a19] /usr/local/lib/glusterfs/1.3.0/xlator/cluster/afr.so[0x2aaaab5100c8] /usr/local/lib/glusterfs/1.3.0/xlator/cluster/unify.so(unify_writev+0xf7)[0x2aaaab722097] /usr/local/lib/glusterfs/1.3.0/xlator/performance/io-threads.so[0x2aaaab92c873] /usr/local/lib/libglusterfs.so.0[0x2aaaaaad7df4] /usr/local/lib/libglusterfs.so.0(call_resume+0x6a)[0x2aaaaaad826a] /usr/local/lib/glusterfs/1.3.0/xlator/performance/io-threads.so[0x2aaaab92dded] /lib64/libpthread.so.0[0x3f610062f7] /lib64/libc.so.6(clone+0x6d)[0x3f604ce86d] --------- ><> Nathan Stratton CTO, Voila IP Communications nathan at robotics.net nathan at voilaip.com http://www.robotics.net http://www.voilaip.com