I set the timeout in the client and server vol files but I'm still running into the same problem. The only difference now is I can still access the file system. -Peter 2008-08-08 13:58:56 W [client-protocol.c:205:call_bail] client-stripe-2: activating bail-out. pending frames = 1. last sent = 2008-08-08 13:58:44. last received = 2008-08-08 13:58:38 transport-timeout = 10 2008-08-08 13:58:56 C [client-protocol.c:212:call_bail] client-stripe-2: bailing transport 2008-08-08 13:58:56 W [client-protocol.c:4777:client_protocol_cleanup] client-stripe-2: cleaning up state in transport object 0x1d8bb560 2008-08-08 13:58:56 E [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: forced unwinding frame type(1) op(14) reply=@0x1d8c4010 2008-08-08 13:58:56 E [client-protocol.c:3247:client_write_cbk] client-stripe-2: no proper reply from server, returning ENOTCONN 2008-08-08 13:58:56 W [stripe.c:3070:stripe_writev_cbk] stripe: client-stripe-2 returned errno 107 2008-08-08 13:58:56 E [fuse-bridge.c:1645:fuse_writev_cbk] glusterfs-fuse: 87006: WRITE => -1 (107) 2008-08-08 13:58:56 W [client-protocol.c:280:client_protocol_xfer] client-stripe-2: attempting to pipeline request type(1) op(34) with handshake 2008-08-08 13:59:16 W [client-protocol.c:205:call_bail] client-stripe-2: activating bail-out. pending frames = 1. last sent = 2008-08-08 13:58:56. last received = 1969-12-31 19:00:00 transport-timeout = 10 2008-08-08 13:59:16 C [client-protocol.c:212:call_bail] client-stripe-2: bailing transport 2008-08-08 13:59:16 W [client-protocol.c:4777:client_protocol_cleanup] client-stripe-2: cleaning up state in transport object 0x1d8bb560 2008-08-08 13:59:16 E [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: forced unwinding frame type(1) op(34) reply=@0x1d8c4010 2008-08-08 13:59:16 E [client-protocol.c:4423:client_lookup_cbk] client-stripe-2: no proper reply from server, returning ENOTCONN 2008-08-08 13:59:16 W [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: client-stripe-2 returned errno 107 2008-08-08 13:59:16 E [fuse-bridge.c:468:fuse_entry_cbk] glusterfs-fuse: 87007: (34) /iozone.DUMMY.4 => -1 (107) 2008-08-08 13:59:16 W [client-protocol.c:280:client_protocol_xfer] client-stripe-2: attempting to pipeline request type(1) op(34) with handshake 2008-08-08 13:59:38 W [client-protocol.c:205:call_bail] client-stripe-2: activating bail-out. pending frames = 1. last sent = 2008-08-08 13:59:16. last received = 1969-12-31 19:00:00 transport-timeout = 10 2008-08-08 13:59:38 C [client-protocol.c:212:call_bail] client-stripe-2: bailing transport 2008-08-08 13:59:38 W [client-protocol.c:4777:client_protocol_cleanup] client-stripe-2: cleaning up state in transport object 0x1d8bb560 2008-08-08 13:59:38 E [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: forced unwinding frame type(1) op(34) reply=@0x1d8c70a0 2008-08-08 13:59:38 E [client-protocol.c:4423:client_lookup_cbk] client-stripe-2: no proper reply from server, returning ENOTCONN 2008-08-08 13:59:38 W [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: client-stripe-2 returned errno 107 2008-08-08 13:59:38 E [fuse-bridge.c:468:fuse_entry_cbk] glusterfs-fuse: 87007: (34) /iozone.DUMMY.4 => -1 (107) 2008-08-08 13:59:38 W [client-protocol.c:280:client_protocol_xfer] client-stripe-2: attempting to pipeline request type(1) op(34) with handshake 2008-08-08 13:59:38 E [client-protocol.c:1238:client_flush] client-stripe-2: : returning EBADFD 2008-08-08 13:59:38 W [stripe.c:198:stripe_stack_unwind_cbk] stripe: client-stripe-2 returned errno 77 2008-08-08 13:59:38 W [client-protocol.c:1296:client_close] client-stripe-2: no valid fd found, returning Anand Babu Periasamy wrote: > Hi Peter, > You need to adjust your time out settings. 1.4.x already addresses > these issues using non-blocking I/O. Currently 1.4.x code cleanup is > going > on, which is leading to rocky road once again, but it is completely > worth. > A month from now, 1.4.0 will be all set. > > -- > Anand Babu Periasamy > GPG Key ID: 0x62E15A31 > Blog [http://ab.freeshell.org] > The GNU Operating System [http://www.gnu.org] > Z RESEARCH Inc [http://www.zresearch.com] > > > > Peter Pelekis wrote: >> I'm having some problems with my glusterfs file system when I run >> benchmarks on it. I'm running iozone in throughput mode and I keep >> losing the glusterfs mount point on random machines. I get the error >> "Transport endpoint is not connected" when I try and run df or ls on the >> node it failed on. The one thing I did notice this morning one of the >> systems that fail last night started working again when I got in this >> morning. The failures I'm seeing are at the end of the log files. >> System: >> 4 Storage nodes strip >> 46 compute nodes >> Centos 5.2 >> 2.6.18-53.el5 >> Glusterfs 1.3.10 >> Fuse fuse-2.7.3glfs10 >> Qlogic QLE7240 IB cards >> >> Heres some of the messages I'm seeing in the log files from the nodes >> that are mounting the file system. >> >> 2008-08-07 19:46:22 W [client-protocol.c:205:call_bail] >> client-stripe-1: activating bail-out. pending frames = 1. last sent = >> 2008-08-07 19:45:32. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:46:22 C [client-protocol.c:212:call_bail] >> client-stripe-1: bailing transport >> 2008-08-07 19:46:22 W [client-protocol.c:205:call_bail] >> client-stripe-4: activating bail-out. pending frames = 1. last sent = >> 2008-08-07 19:45:32. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:46:22 C [client-protocol.c:212:call_bail] >> client-stripe-4: bailing transport >> 2008-08-07 19:46:22 W [client-protocol.c:205:call_bail] >> client-stripe-2: activating bail-out. pending frames = 1. last sent = >> 2008-08-07 19:45:32. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:46:22 C [client-protocol.c:212:call_bail] >> client-stripe-2: bailing transport >> 2008-08-07 19:46:22 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-2: >> cleaning up state in transport object 0x9625260 >> 2008-08-07 19:46:22 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: >> forced unwinding frame type(1) op(34) reply=@0x962f1f0 >> 2008-08-07 19:46:22 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-2: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:46:22 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-2 returned errno 107 >> 2008-08-07 19:46:22 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-4: >> cleaning up state in transport object 0x9629f20 >> 2008-08-07 19:46:22 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: >> forced unwinding frame type(1) op(34) reply=@0x962e190 >> 2008-08-07 19:46:22 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-4: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:46:22 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-4 returned errno 107 >> 2008-08-07 19:46:22 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-1: >> cleaning up state in transport object 0x961f350 >> 2008-08-07 19:46:22 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: >> forced unwinding frame type(1) op(34) reply=@0x9627630 >> 2008-08-07 19:46:22 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-1: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:46:22 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-1 returned errno 107 >> 2008-08-07 19:47:25 W [client-protocol.c:205:call_bail] >> client-stripe-3: activating bail-out. pending frames = 1. last sent = >> 2008-08-07 19:45:32. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:47:25 C [client-protocol.c:212:call_bail] >> client-stripe-3: bailing transport >> 2008-08-07 19:47:25 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-3: >> cleaning up state in transport object 0x96278c0 >> 2008-08-07 19:47:25 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: >> forced unwinding frame type(1) op(34) reply=@0x962c2f0 >> 2008-08-07 19:47:25 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-3: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:47:25 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-3 returned errno 107 >> 2008-08-07 19:47:25 E [fuse-bridge.c:468:fuse_entry_cbk] >> glusterfs-fuse: 63760: (34) / => -1 (107) >> 2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] >> trans: attempting to pipeline request type(2) op(4) with handshake >> 2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-1: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-2: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-3: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-4: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:53:23 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-1: attempting to pipeline request type(1) op(15) with >> handshake >> 2008-08-07 19:53:23 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-2: attempting to pipeline request type(1) op(15) with >> handshake >> 2008-08-07 19:53:23 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-3: attempting to pipeline request type(1) op(15) with >> handshake >> 2008-08-07 19:53:23 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-4: attempting to pipeline request type(1) op(15) with >> handshake >> 2008-08-07 19:54:06 W [client-protocol.c:205:call_bail] >> client-stripe-1: activating bail-out. pending frames = 2. last sent = >> 2008-08-07 19:53:23. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:54:06 C [client-protocol.c:212:call_bail] >> client-stripe-1: bailing transport >> 2008-08-07 19:54:06 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-1: >> cleaning up state in transport object 0x961f350 >> 2008-08-07 19:54:06 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: >> forced unwinding frame type(1) op(34) reply=@0x9629c90 >> 2008-08-07 19:54:06 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-1: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:54:06 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-1 returned errno 107 >> 2008-08-07 19:54:06 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: >> forced unwinding frame type(1) op(15) reply=@0x9629c90 >> 2008-08-07 19:54:06 E [client-protocol.c:3884:client_statfs_cbk] >> client-stripe-1: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:54:09 W [client-protocol.c:205:call_bail] >> client-stripe-4: activating bail-out. pending frames = 2. last sent = >> 2008-08-07 19:53:23. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:54:09 C [client-protocol.c:212:call_bail] >> client-stripe-4: bailing transport >> 2008-08-07 19:54:09 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-4: >> cleaning up state in transport object 0x9629f20 >> 2008-08-07 19:54:09 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: >> forced unwinding frame type(1) op(34) reply=@0x962e190 >> 2008-08-07 19:54:09 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-4: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:54:09 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-4 returned errno 107 >> 2008-08-07 19:54:09 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: >> forced unwinding frame type(1) op(15) reply=@0x962e190 >> 2008-08-07 19:54:09 E [client-protocol.c:3884:client_statfs_cbk] >> client-stripe-4: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:54:10 W [client-protocol.c:205:call_bail] >> client-stripe-3: activating bail-out. pending frames = 2. last sent = >> 2008-08-07 19:53:23. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:54:10 C [client-protocol.c:212:call_bail] >> client-stripe-3: bailing transport >> 2008-08-07 19:54:10 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-3: >> cleaning up state in transport object 0x96278c0 >> 2008-08-07 19:54:10 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: >> forced unwinding frame type(1) op(34) reply=@0x9627630 >> 2008-08-07 19:54:10 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-3: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:54:10 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-3 returned errno 107 >> 2008-08-07 19:54:10 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: >> forced unwinding frame type(1) op(15) reply=@0x9627630 >> 2008-08-07 19:54:10 E [client-protocol.c:3884:client_statfs_cbk] >> client-stripe-3: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:54:13 W [client-protocol.c:205:call_bail] >> client-stripe-2: activating bail-out. pending frames = 2. last sent = >> 2008-08-07 19:53:23. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:54:13 C [client-protocol.c:212:call_bail] >> client-stripe-2: bailing transport >> 2008-08-07 19:54:13 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-2: >> cleaning up state in transport object 0x9625260 >> 2008-08-07 19:54:13 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: >> forced unwinding frame type(1) op(34) reply=@0x962c870 >> 2008-08-07 19:54:13 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-2: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:54:13 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-2 returned errno 107 >> 2008-08-07 19:54:13 E [fuse-bridge.c:468:fuse_entry_cbk] >> glusterfs-fuse: 63761: (34) / => -1 (107) >> 2008-08-07 19:54:13 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-1: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:54:13 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-2: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:54:13 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-3: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:54:13 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-4: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:54:13 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: >> forced unwinding frame type(1) op(15) reply=@0x962c870 >> 2008-08-07 19:54:13 E [client-protocol.c:3884:client_statfs_cbk] >> client-stripe-2: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:54:13 E [fuse-bridge.c:2064:fuse_statfs_cbk] >> glusterfs-fuse: 63762: ERR => -1 (107) >> 2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] >> trans: attempting to pipeline request type(2) op(4) with handshake >> 2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-1: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-2: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-3: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-4: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:55:14 W [client-protocol.c:205:call_bail] >> client-stripe-3: activating bail-out. pending frames = 2. last sent = >> 2008-08-07 19:54:31. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:55:14 C [client-protocol.c:212:call_bail] >> client-stripe-3: bailing transport >> 2008-08-07 19:55:14 W [client-protocol.c:205:call_bail] >> client-stripe-4: activating bail-out. pending frames = 2. last sent = >> 2008-08-07 19:54:31. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:55:14 C [client-protocol.c:212:call_bail] >> client-stripe-4: bailing transport >> 2008-08-07 19:55:14 W [client-protocol.c:205:call_bail] >> client-stripe-1: activating bail-out. pending frames = 2. last sent = >> 2008-08-07 19:54:31. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:55:14 C [client-protocol.c:212:call_bail] >> client-stripe-1: bailing transport >> 2008-08-07 19:55:14 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-1: >> cleaning up state in transport object 0x961f350 >> 2008-08-07 19:55:14 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: >> forced unwinding frame type(1) op(34) reply=@0x9629c90 >> 2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-1: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:55:14 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-1 returned errno 107 >> 2008-08-07 19:55:14 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: >> forced unwinding frame type(1) op(34) reply=@0x9629c90 >> 2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-1: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:55:14 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-1 returned errno 107 >> 2008-08-07 19:55:14 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-4: >> cleaning up state in transport object 0x9629f20 >> 2008-08-07 19:55:14 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: >> forced unwinding frame type(1) op(34) reply=@0x962c870 >> 2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-4: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:55:14 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-4 returned errno 107 >> 2008-08-07 19:55:14 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: >> forced unwinding frame type(1) op(34) reply=@0x962c870 >> 2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-4: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:55:14 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-4 returned errno 107 >> 2008-08-07 19:55:14 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-3: >> cleaning up state in transport object 0x96278c0 >> 2008-08-07 19:55:14 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: >> forced unwinding frame type(1) op(34) reply=@0x9627630 >> 2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-3: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:55:14 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-3 returned errno 107 >> 2008-08-07 19:55:14 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: >> forced unwinding frame type(1) op(34) reply=@0x9627630 >> 2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-3: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:55:14 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-3 returned errno 107 >> 2008-08-07 19:55:21 W [client-protocol.c:205:call_bail] >> client-stripe-2: activating bail-out. pending frames = 2. last sent = >> 2008-08-07 19:54:31. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:55:21 C [client-protocol.c:212:call_bail] >> client-stripe-2: bailing transport >> 2008-08-07 19:55:21 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-2: >> cleaning up state in transport object 0x9625260 >> 2008-08-07 19:55:21 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: >> forced unwinding frame type(1) op(34) reply=@0x962e3b0 >> 2008-08-07 19:55:21 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-2: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:55:21 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-2 returned errno 107 >> 2008-08-07 19:55:21 E [fuse-bridge.c:468:fuse_entry_cbk] >> glusterfs-fuse: 63761: (34) / => -1 (107) >> 2008-08-07 19:55:21 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: >> forced unwinding frame type(1) op(34) reply=@0x962e3b0 >> 2008-08-07 19:55:21 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-2: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:55:21 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-2 returned errno 107 >> 2008-08-07 19:55:21 E [fuse-bridge.c:468:fuse_entry_cbk] >> glusterfs-fuse: 63763: (34) / => -1 (107) >> 2008-08-07 19:55:21 E [fuse-bridge.c:2699:init] glusterfs-fuse: >> fuse_mount failed (Transport endpoint is not connected) >> >> 2008-08-07 19:55:21 E [glusterfs.c:547:main] glusterfs: Initializing >> FUSE failed >> 2008-08-07 19:55:21 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-1: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:55:21 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-2: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:55:21 E [ib-verbs.c:1107:ib_verbs_send_completion_proc] >> transport/ib-verbs: send work request on `ipath0' returned error >> wc.status = 12, wc.vendor_err = 0, post->buf = 0x2aaaac530000, >> wc.byte_len = 0, post->reused = 35 >> 2008-08-07 19:55:21 E [ib-verbs.c:1500:ib_verbs_except] >> transport/ib-verbs: shutdown () - error: Transport endpoint is not >> connected >> 2008-08-07 19:55:24 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-3: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:55:24 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-4: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:55:45 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-1: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:55:45 E [ib-verbs-client.c:141:ib_verbs_client_connect] >> client-stripe-2: connection not in progress - trace: Network is >> unreachable >> 2008-08-07 19:55:45 W [client-protocol.c:332:client_protocol_xfer] >> client-stripe-2: not connected at the moment to submit frame type(1) >> op(34) >> 2008-08-07 19:55:45 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-2: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:55:45 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-2 returned errno 107 >> 2008-08-07 19:55:45 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-3: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:55:45 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-4: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:56:32 W [client-protocol.c:205:call_bail] >> client-stripe-1: activating bail-out. pending frames = 2. last sent = >> 2008-08-07 19:55:45. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:56:32 C [client-protocol.c:212:call_bail] >> client-stripe-1: bailing transport >> 2008-08-07 19:56:32 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-1: >> cleaning up state in transport object 0x961f350 >> 2008-08-07 19:56:32 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: >> forced unwinding frame type(1) op(34) reply=@0x962f690 >> 2008-08-07 19:56:32 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-1: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:56:32 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-1 returned errno 107 >> 2008-08-07 19:56:32 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: >> forced unwinding frame type(1) op(34) reply=@0x962f690 >> 2008-08-07 19:56:32 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-1: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:56:32 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-1 returned errno 107 >> 2008-08-07 19:56:35 W [client-protocol.c:205:call_bail] >> client-stripe-4: activating bail-out. pending frames = 2. last sent = >> 2008-08-07 19:55:45. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:56:35 C [client-protocol.c:212:call_bail] >> client-stripe-4: bailing transport >> 2008-08-07 19:56:35 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-4: >> cleaning up state in transport object 0x9629f20 >> 2008-08-07 19:56:35 W [client-protocol.c:205:call_bail] >> client-stripe-3: activating bail-out. pending frames = 2. last sent = >> 2008-08-07 19:55:45. last received = 1969-12-31 19:00:00 >> transport-timeout = 42 >> 2008-08-07 19:56:35 C [client-protocol.c:212:call_bail] >> client-stripe-3: bailing transport >> 2008-08-07 19:56:35 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: >> forced unwinding frame type(1) op(34) reply=@0x962c870 >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-4: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:56:35 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-4 returned errno 107 >> 2008-08-07 19:56:35 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: >> forced unwinding frame type(1) op(34) reply=@0x962c870 >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-4: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:56:35 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-4 returned errno 107 >> 2008-08-07 19:56:35 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-3: >> cleaning up state in transport object 0x96278c0 >> 2008-08-07 19:56:35 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: >> forced unwinding frame type(1) op(34) reply=@0x9629c90 >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-3: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:56:35 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-3 returned errno 107 >> 2008-08-07 19:56:35 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: >> forced unwinding frame type(1) op(34) reply=@0x9629c90 >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-3: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:56:35 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-3 returned errno 107 >> 2008-08-07 19:56:35 E [fuse-bridge.c:468:fuse_entry_cbk] >> glusterfs-fuse: 63764: (34) / => -1 (107) >> 2008-08-07 19:56:35 E [ib-verbs-client.c:141:ib_verbs_client_connect] >> client-stripe-1: connection not in progress - trace: Network is >> unreachable >> 2008-08-07 19:56:35 W [client-protocol.c:332:client_protocol_xfer] >> client-stripe-1: not connected at the moment to submit frame type(1) >> op(34) >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-1: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:56:35 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-1 returned errno 107 >> 2008-08-07 19:56:35 E [ib-verbs-client.c:141:ib_verbs_client_connect] >> client-stripe-2: connection not in progress - trace: Network is >> unreachable >> 2008-08-07 19:56:35 W [client-protocol.c:332:client_protocol_xfer] >> client-stripe-2: not connected at the moment to submit frame type(1) >> op(34) >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-2: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:56:35 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-2 returned errno 107 >> 2008-08-07 19:56:35 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-3: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 19:56:35 E [ib-verbs-client.c:141:ib_verbs_client_connect] >> client-stripe-4: connection not in progress - trace: Network is >> unreachable >> 2008-08-07 19:56:35 W [client-protocol.c:332:client_protocol_xfer] >> client-stripe-4: not connected at the moment to submit frame type(1) >> op(34) >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-4: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:56:35 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-4 returned errno 107 >> 2008-08-07 19:56:45 E [ib-verbs-client.c:141:ib_verbs_client_connect] >> client-stripe-4: connection not in progress - trace: Network is >> unreachable >> 2008-08-07 19:57:25 E [ib-verbs-client.c:141:ib_verbs_client_connect] >> client-stripe-3: connection not in progress - trace: Network is >> unreachable >> 2008-08-07 19:57:28 E [ib-verbs-client.c:141:ib_verbs_client_connect] >> client-stripe-1: connection not in progress - trace: Network is >> unreachable >> 2008-08-07 19:57:28 W [client-protocol.c:332:client_protocol_xfer] >> client-stripe-1: not connected at the moment to submit frame type(1) >> op(34) >> 2008-08-07 19:57:28 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-1: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:57:28 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-1 returned errno 107 >> 2008-08-07 19:57:28 E [ib-verbs-client.c:141:ib_verbs_client_connect] >> client-stripe-2: connection not in progress - trace: Network is >> unreachable >> 2008-08-07 19:57:28 W [client-protocol.c:332:client_protocol_xfer] >> client-stripe-2: not connected at the moment to submit frame type(1) >> op(34) >> 2008-08-07 19:57:28 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-2: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:57:28 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-2 returned errno 107 >> 2008-08-07 19:57:28 E [ib-verbs-client.c:141:ib_verbs_client_connect] >> client-stripe-3: connection not in progress - trace: Network is >> unreachable >> 2008-08-07 19:57:28 W [client-protocol.c:332:client_protocol_xfer] >> client-stripe-3: not connected at the moment to submit frame type(1) >> op(34) >> 2008-08-07 19:57:28 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-3: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:57:28 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-3 returned errno 107 >> 2008-08-07 19:57:28 E [ib-verbs-client.c:141:ib_verbs_client_connect] >> client-stripe-4: connection not in progress - trace: Network is >> unreachable >> 2008-08-07 19:57:28 W [client-protocol.c:332:client_protocol_xfer] >> client-stripe-4: not connected at the moment to submit frame type(1) >> op(34) >> 2008-08-07 19:57:28 E [client-protocol.c:4423:client_lookup_cbk] >> client-stripe-4: no proper reply from server, returning ENOTCONN >> 2008-08-07 19:57:28 W >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: >> client-stripe-4 returned errno 107 >> 2008-08-07 19:57:28 E [fuse-bridge.c:468:fuse_entry_cbk] >> glusterfs-fuse: 63765: (34) /iozone.DUMMY.14 => -1 (107) >> 2008-08-07 19:57:29 E [ib-verbs-client.c:141:ib_verbs_client_connect] >> client-stripe-2: connection not in progress - trace: Network is >> unreachable >> 2008-08-07 19:57:30 W [client-protocol.c:1296:client_close] >> client-stripe-2: no valid fd found, returning >> 2008-08-07 19:57:30 W [client-protocol.c:1296:client_close] >> client-stripe-3: no valid fd found, returning >> 2008-08-07 19:57:30 W [client-protocol.c:1296:client_close] >> client-stripe-4: no valid fd found, returning >> 2008-08-07 19:57:30 W [client-protocol.c:1296:client_close] >> client-stripe-1: no valid fd found, returning >> 2008-08-07 19:57:33 E [ib-verbs-client.c:141:ib_verbs_client_connect] >> client-stripe-1: connection not in progress - trace: Network is >> unreachable >> 2008-08-07 20:00:04 W [client-protocol.c:280:client_protocol_xfer] >> trans: attempting to pipeline request type(2) op(4) with handshake >> 2008-08-07 20:00:04 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-2: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 20:00:04 W [fuse-bridge.c:398:fuse_entry_cbk] >> glusterfs-fuse: 2: (34) / => 1 Rehashing 0/0 >> 2008-08-07 23:31:29 W [client-protocol.c:280:client_protocol_xfer] >> trans: attempting to pipeline request type(2) op(4) with handshake >> 2008-08-07 23:31:29 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-3: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 23:31:29 W [fuse-bridge.c:398:fuse_entry_cbk] >> glusterfs-fuse: 2: (34) / => 1 Rehashing 0/0 >> 2008-08-07 23:59:06 W [client-protocol.c:205:call_bail] >> client-stripe-2: activating bail-out. pending frames = 1. last sent = >> 2008-08-07 23:58:23. last received = 2008-08-07 23:58:17 >> transport-timeout = 42 >> 2008-08-07 23:59:06 C [client-protocol.c:212:call_bail] >> client-stripe-2: bailing transport >> 2008-08-07 23:59:06 W >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-2: >> cleaning up state in transport object 0x87eb260 >> 2008-08-07 23:59:06 E >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: >> forced unwinding frame type(1) op(14) reply=@0x87f3d10 >> 2008-08-07 23:59:06 E [client-protocol.c:3247:client_write_cbk] >> client-stripe-2: no proper reply from server, returning ENOTCONN >> 2008-08-07 23:59:06 W [stripe.c:3070:stripe_writev_cbk] stripe: >> client-stripe-2 returned errno 107 >> 2008-08-07 23:59:06 E [fuse-bridge.c:1645:fuse_writev_cbk] >> glusterfs-fuse: 128670: WRITE => -1 (107) >> 2008-08-07 23:59:06 W [client-protocol.c:280:client_protocol_xfer] >> client-stripe-2: attempting to pipeline request type(1) op(34) with >> handshake >> 2008-08-07 23:59:28 E [client-protocol.c:1238:client_flush] >> client-stripe-2: : returning EBADFD >> 2008-08-07 23:59:28 W [stripe.c:198:stripe_stack_unwind_cbk] stripe: >> client-stripe-2 returned errno 77 >> 2008-08-07 23:59:28 W [client-protocol.c:1296:client_close] >> client-stripe-2: no valid fd found, returning >> 2008-08-08 00:51:00 W [client-protocol.c:1296:client_close] >> client-stripe-2: no valid fd found, returning >> >> >> -Peter >> -- Peter Pelekis 5444 Napa Street Western Scientific San Diego CA, 92110 Senior Systems Engineer www.wsm.com pete.pelekis at wsm.com Fax 619-220-6590 Phone 619-220-6580 x212 Toll Free 800-443-6699 GSA# GS-35F-5009H Visit us at: "I'm a great believer in luck, and I find the harder I work the more I have of it." -Thomas Jefferson "If everything seems under control, you're not going fast enough." -Mario Andretti
