Amar Its a basic strip setup across 4 storage servers. I can send you the vol files if you would like to see them. -Peter Amar S. Tumballi wrote: > Peter, > Sorry for the delay.. I would be able to test ib-verbs next week. > Will get back to you mostly on monday. > > Is it simple stripe setup? > > Regards, > Amar > > 2008/8/8 Peter Pelekis <pete.pelekis at wsm.com > <mailto:pete.pelekis at wsm.com>> > > Heres some of the errors I'm seeing on the storage servers. > > -Peter > > 2008-08-08 13:59:33 C [ib-verbs-server.c:233:gf_transport_fini] > ib-verbs/server: server: called fini on transport: 0x16633d10 > 2008-08-08 13:59:37 C [ib-verbs.c:1550:ib_verbs_tcp_notify] > transport/ib-verbs: server: notify (2) called on tcp socket > 2008-08-08 13:59:37 C [ib-verbs-server.c:233:gf_transport_fini] > ib-verbs/server: server: called fini on transport: 0x1662f740 > 2008-08-08 13:59:38 C [ib-verbs.c:1550:ib_verbs_tcp_notify] > transport/ib-verbs: server: notify (2) called on tcp socket > 2008-08-08 13:59:38 C [ib-verbs-server.c:233:gf_transport_fini] > ib-verbs/server: server: called fini on transport: 0x16626690 > 2008-08-08 13:59:39 C [ib-verbs.c:1550:ib_verbs_tcp_notify] > transport/ib-verbs: server: notify (2) called on tcp socket > 2008-08-08 13:59:39 C [ib-verbs-server.c:233:gf_transport_fini] > ib-verbs/server: server: called fini on transport: 0x1662e470 > 2008-08-08 13:59:41 C [ib-verbs.c:1550:ib_verbs_tcp_notify] > transport/ib-verbs: server: notify (2) called on tcp socket > 2008-08-08 13:59:41 C [ib-verbs-server.c:233:gf_transport_fini] > ib-verbs/server: server: called fini on transport: 0x16625080 > 2008-08-08 13:59:44 C [ib-verbs.c:1550:ib_verbs_tcp_notify] > transport/ib-verbs: server: notify (2) called on tcp socket > 2008-08-08 13:59:44 C [ib-verbs-server.c:233:gf_transport_fini] > ib-verbs/server: server: called fini on transport: 0x16624b40 > 2008-08-08 13:59:46 C [ib-verbs.c:1550:ib_verbs_tcp_notify] > transport/ib-verbs: server: notify (2) called on tcp socket > 2008-08-08 13:59:46 C [ib-verbs-server.c:233:gf_transport_fini] > ib-verbs/server: server: called fini on transport: 0x1663a120 > 2008-08-08 13:59:46 C [ib-verbs.c:1550:ib_verbs_tcp_notify] > transport/ib-verbs: server: notify (2) called on tcp socket > 2008-08-08 13:59:46 C [ib-verbs-server.c:233:gf_transport_fini] > ib-verbs/server: server: called fini on transport: 0x16623080 > 2008-08-08 14:00:00 C [ib-verbs.c:1550:ib_verbs_tcp_notify] > transport/ib-verbs: server: notify (2) called on tcp socket > 2008-08-08 14:00:00 C [ib-verbs-server.c:233:gf_transport_fini] > ib-verbs/server: server: called fini on transport: 0x1662e470 > 2008-08-08 14:00:02 C [ib-verbs.c:1550:ib_verbs_tcp_notify] > transport/ib-verbs: server: notify (2) called on tcp socket > 2008-08-08 14:00:02 C [ib-verbs-server.c:233:gf_transport_fini] > ib-verbs/server: server: called fini on transport: 0x16625080 > 2008-08-08 14:00:05 C [ib-verbs.c:1550:ib_verbs_tcp_notify] > transport/ib-verbs: server: notify (2) called on tcp socket > 2008-08-08 14:00:05 C [ib-verbs-server.c:233:gf_transport_fini] > ib-verbs/server: server: called fini on transport: 0x16624b40 > 2008-08-08 14:00:07 C [ib-verbs.c:1550:ib_verbs_tcp_notify] > transport/ib-verbs: server: notify (2) called on tcp socket > 2008-08-08 14:00:07 C [ib-verbs-server.c:233:gf_transport_fini] > ib-verbs/server: server: called fini on transport: 0x16626690 > > > Anand Babu Periasamy wrote: > > Hi Peter, > > You need to adjust your time out settings. 1.4.x already addresses > > these issues using non-blocking I/O. Currently 1.4.x code cleanup is > > going > > on, which is leading to rocky road once again, but it is completely > > worth. > > A month from now, 1.4.0 will be all set. > > > > -- > > Anand Babu Periasamy > > GPG Key ID: 0x62E15A31 > > Blog [http://ab.freeshell.org] > > The GNU Operating System [http://www.gnu.org] > > Z RESEARCH Inc [http://www.zresearch.com] > > > > > > > > Peter Pelekis wrote: > >> I'm having some problems with my glusterfs file system when I run > >> benchmarks on it. I'm running iozone in throughput mode and I keep > >> losing the glusterfs mount point on random machines. I get the > error > >> "Transport endpoint is not connected" when I try and run df or > ls on the > >> node it failed on. The one thing I did notice this morning one > of the > >> systems that fail last night started working again when I got > in this > >> morning. The failures I'm seeing are at the end of the log files. > >> System: > >> 4 Storage nodes strip > >> 46 compute nodes > >> Centos 5.2 > >> 2.6.18-53.el5 > >> Glusterfs 1.3.10 > >> Fuse fuse-2.7.3glfs10 > >> Qlogic QLE7240 IB cards > >> > >> Heres some of the messages I'm seeing in the log files from the > nodes > >> that are mounting the file system. > >> > >> 2008-08-07 19:46:22 W [client-protocol.c:205:call_bail] > >> client-stripe-1: activating bail-out. pending frames = 1. last > sent = > >> 2008-08-07 19:45:32. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:46:22 C [client-protocol.c:212:call_bail] > >> client-stripe-1: bailing transport > >> 2008-08-07 19:46:22 W [client-protocol.c:205:call_bail] > >> client-stripe-4: activating bail-out. pending frames = 1. last > sent = > >> 2008-08-07 19:45:32. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:46:22 C [client-protocol.c:212:call_bail] > >> client-stripe-4: bailing transport > >> 2008-08-07 19:46:22 W [client-protocol.c:205:call_bail] > >> client-stripe-2: activating bail-out. pending frames = 1. last > sent = > >> 2008-08-07 19:45:32. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:46:22 C [client-protocol.c:212:call_bail] > >> client-stripe-2: bailing transport > >> 2008-08-07 19:46:22 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-2: > >> cleaning up state in transport object 0x9625260 > >> 2008-08-07 19:46:22 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: > >> forced unwinding frame type(1) op(34) reply=@0x962f1f0 > >> 2008-08-07 19:46:22 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-2: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:46:22 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-2 returned errno 107 > >> 2008-08-07 19:46:22 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-4: > >> cleaning up state in transport object 0x9629f20 > >> 2008-08-07 19:46:22 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: > >> forced unwinding frame type(1) op(34) reply=@0x962e190 > >> 2008-08-07 19:46:22 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-4: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:46:22 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-4 returned errno 107 > >> 2008-08-07 19:46:22 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-1: > >> cleaning up state in transport object 0x961f350 > >> 2008-08-07 19:46:22 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: > >> forced unwinding frame type(1) op(34) reply=@0x9627630 > >> 2008-08-07 19:46:22 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-1: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:46:22 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-1 returned errno 107 > >> 2008-08-07 19:47:25 W [client-protocol.c:205:call_bail] > >> client-stripe-3: activating bail-out. pending frames = 1. last > sent = > >> 2008-08-07 19:45:32. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:47:25 C [client-protocol.c:212:call_bail] > >> client-stripe-3: bailing transport > >> 2008-08-07 19:47:25 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-3: > >> cleaning up state in transport object 0x96278c0 > >> 2008-08-07 19:47:25 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: > >> forced unwinding frame type(1) op(34) reply=@0x962c2f0 > >> 2008-08-07 19:47:25 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-3: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:47:25 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-3 returned errno 107 > >> 2008-08-07 19:47:25 E [fuse-bridge.c:468:fuse_entry_cbk] > >> glusterfs-fuse: 63760: (34) / => -1 (107) > >> 2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] > >> trans: attempting to pipeline request type(2) op(4) with handshake > >> 2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-1: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-2: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-3: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:53:20 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-4: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:53:23 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-1: attempting to pipeline request type(1) op(15) with > >> handshake > >> 2008-08-07 19:53:23 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-2: attempting to pipeline request type(1) op(15) with > >> handshake > >> 2008-08-07 19:53:23 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-3: attempting to pipeline request type(1) op(15) with > >> handshake > >> 2008-08-07 19:53:23 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-4: attempting to pipeline request type(1) op(15) with > >> handshake > >> 2008-08-07 19:54:06 W [client-protocol.c:205:call_bail] > >> client-stripe-1: activating bail-out. pending frames = 2. last > sent = > >> 2008-08-07 19:53:23. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:54:06 C [client-protocol.c:212:call_bail] > >> client-stripe-1: bailing transport > >> 2008-08-07 19:54:06 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-1: > >> cleaning up state in transport object 0x961f350 > >> 2008-08-07 19:54:06 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: > >> forced unwinding frame type(1) op(34) reply=@0x9629c90 > >> 2008-08-07 19:54:06 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-1: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:54:06 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-1 returned errno 107 > >> 2008-08-07 19:54:06 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: > >> forced unwinding frame type(1) op(15) reply=@0x9629c90 > >> 2008-08-07 19:54:06 E [client-protocol.c:3884:client_statfs_cbk] > >> client-stripe-1: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:54:09 W [client-protocol.c:205:call_bail] > >> client-stripe-4: activating bail-out. pending frames = 2. last > sent = > >> 2008-08-07 19:53:23. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:54:09 C [client-protocol.c:212:call_bail] > >> client-stripe-4: bailing transport > >> 2008-08-07 19:54:09 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-4: > >> cleaning up state in transport object 0x9629f20 > >> 2008-08-07 19:54:09 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: > >> forced unwinding frame type(1) op(34) reply=@0x962e190 > >> 2008-08-07 19:54:09 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-4: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:54:09 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-4 returned errno 107 > >> 2008-08-07 19:54:09 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: > >> forced unwinding frame type(1) op(15) reply=@0x962e190 > >> 2008-08-07 19:54:09 E [client-protocol.c:3884:client_statfs_cbk] > >> client-stripe-4: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:54:10 W [client-protocol.c:205:call_bail] > >> client-stripe-3: activating bail-out. pending frames = 2. last > sent = > >> 2008-08-07 19:53:23. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:54:10 C [client-protocol.c:212:call_bail] > >> client-stripe-3: bailing transport > >> 2008-08-07 19:54:10 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-3: > >> cleaning up state in transport object 0x96278c0 > >> 2008-08-07 19:54:10 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: > >> forced unwinding frame type(1) op(34) reply=@0x9627630 > >> 2008-08-07 19:54:10 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-3: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:54:10 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-3 returned errno 107 > >> 2008-08-07 19:54:10 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: > >> forced unwinding frame type(1) op(15) reply=@0x9627630 > >> 2008-08-07 19:54:10 E [client-protocol.c:3884:client_statfs_cbk] > >> client-stripe-3: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:54:13 W [client-protocol.c:205:call_bail] > >> client-stripe-2: activating bail-out. pending frames = 2. last > sent = > >> 2008-08-07 19:53:23. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:54:13 C [client-protocol.c:212:call_bail] > >> client-stripe-2: bailing transport > >> 2008-08-07 19:54:13 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-2: > >> cleaning up state in transport object 0x9625260 > >> 2008-08-07 19:54:13 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: > >> forced unwinding frame type(1) op(34) reply=@0x962c870 > >> 2008-08-07 19:54:13 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-2: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:54:13 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-2 returned errno 107 > >> 2008-08-07 19:54:13 E [fuse-bridge.c:468:fuse_entry_cbk] > >> glusterfs-fuse: 63761: (34) / => -1 (107) > >> 2008-08-07 19:54:13 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-1: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:54:13 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-2: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:54:13 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-3: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:54:13 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-4: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:54:13 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: > >> forced unwinding frame type(1) op(15) reply=@0x962c870 > >> 2008-08-07 19:54:13 E [client-protocol.c:3884:client_statfs_cbk] > >> client-stripe-2: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:54:13 E [fuse-bridge.c:2064:fuse_statfs_cbk] > >> glusterfs-fuse: 63762: ERR => -1 (107) > >> 2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] > >> trans: attempting to pipeline request type(2) op(4) with handshake > >> 2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-1: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-2: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-3: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:54:31 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-4: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:55:14 W [client-protocol.c:205:call_bail] > >> client-stripe-3: activating bail-out. pending frames = 2. last > sent = > >> 2008-08-07 19:54:31. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:55:14 C [client-protocol.c:212:call_bail] > >> client-stripe-3: bailing transport > >> 2008-08-07 19:55:14 W [client-protocol.c:205:call_bail] > >> client-stripe-4: activating bail-out. pending frames = 2. last > sent = > >> 2008-08-07 19:54:31. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:55:14 C [client-protocol.c:212:call_bail] > >> client-stripe-4: bailing transport > >> 2008-08-07 19:55:14 W [client-protocol.c:205:call_bail] > >> client-stripe-1: activating bail-out. pending frames = 2. last > sent = > >> 2008-08-07 19:54:31. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:55:14 C [client-protocol.c:212:call_bail] > >> client-stripe-1: bailing transport > >> 2008-08-07 19:55:14 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-1: > >> cleaning up state in transport object 0x961f350 > >> 2008-08-07 19:55:14 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: > >> forced unwinding frame type(1) op(34) reply=@0x9629c90 > >> 2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-1: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:55:14 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-1 returned errno 107 > >> 2008-08-07 19:55:14 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: > >> forced unwinding frame type(1) op(34) reply=@0x9629c90 > >> 2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-1: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:55:14 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-1 returned errno 107 > >> 2008-08-07 19:55:14 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-4: > >> cleaning up state in transport object 0x9629f20 > >> 2008-08-07 19:55:14 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: > >> forced unwinding frame type(1) op(34) reply=@0x962c870 > >> 2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-4: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:55:14 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-4 returned errno 107 > >> 2008-08-07 19:55:14 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: > >> forced unwinding frame type(1) op(34) reply=@0x962c870 > >> 2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-4: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:55:14 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-4 returned errno 107 > >> 2008-08-07 19:55:14 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-3: > >> cleaning up state in transport object 0x96278c0 > >> 2008-08-07 19:55:14 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: > >> forced unwinding frame type(1) op(34) reply=@0x9627630 > >> 2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-3: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:55:14 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-3 returned errno 107 > >> 2008-08-07 19:55:14 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: > >> forced unwinding frame type(1) op(34) reply=@0x9627630 > >> 2008-08-07 19:55:14 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-3: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:55:14 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-3 returned errno 107 > >> 2008-08-07 19:55:21 W [client-protocol.c:205:call_bail] > >> client-stripe-2: activating bail-out. pending frames = 2. last > sent = > >> 2008-08-07 19:54:31. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:55:21 C [client-protocol.c:212:call_bail] > >> client-stripe-2: bailing transport > >> 2008-08-07 19:55:21 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-2: > >> cleaning up state in transport object 0x9625260 > >> 2008-08-07 19:55:21 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: > >> forced unwinding frame type(1) op(34) reply=@0x962e3b0 > >> 2008-08-07 19:55:21 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-2: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:55:21 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-2 returned errno 107 > >> 2008-08-07 19:55:21 E [fuse-bridge.c:468:fuse_entry_cbk] > >> glusterfs-fuse: 63761: (34) / => -1 (107) > >> 2008-08-07 19:55:21 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: > >> forced unwinding frame type(1) op(34) reply=@0x962e3b0 > >> 2008-08-07 19:55:21 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-2: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:55:21 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-2 returned errno 107 > >> 2008-08-07 19:55:21 E [fuse-bridge.c:468:fuse_entry_cbk] > >> glusterfs-fuse: 63763: (34) / => -1 (107) > >> 2008-08-07 19:55:21 E [fuse-bridge.c:2699:init] glusterfs-fuse: > >> fuse_mount failed (Transport endpoint is not connected) > >> > >> 2008-08-07 19:55:21 E [glusterfs.c:547:main] glusterfs: > Initializing > >> FUSE failed > >> 2008-08-07 19:55:21 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-1: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:55:21 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-2: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:55:21 E > [ib-verbs.c:1107:ib_verbs_send_completion_proc] > >> transport/ib-verbs: send work request on `ipath0' returned error > >> wc.status = 12, wc.vendor_err = 0, post->buf = 0x2aaaac530000, > >> wc.byte_len = 0, post->reused = 35 > >> 2008-08-07 19:55:21 E [ib-verbs.c:1500:ib_verbs_except] > >> transport/ib-verbs: shutdown () - error: Transport endpoint is not > >> connected > >> 2008-08-07 19:55:24 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-3: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:55:24 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-4: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:55:45 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-1: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:55:45 E > [ib-verbs-client.c:141:ib_verbs_client_connect] > >> client-stripe-2: connection not in progress - trace: Network is > >> unreachable > >> 2008-08-07 19:55:45 W [client-protocol.c:332:client_protocol_xfer] > >> client-stripe-2: not connected at the moment to submit frame > type(1) > >> op(34) > >> 2008-08-07 19:55:45 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-2: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:55:45 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-2 returned errno 107 > >> 2008-08-07 19:55:45 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-3: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:55:45 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-4: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:56:32 W [client-protocol.c:205:call_bail] > >> client-stripe-1: activating bail-out. pending frames = 2. last > sent = > >> 2008-08-07 19:55:45. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:56:32 C [client-protocol.c:212:call_bail] > >> client-stripe-1: bailing transport > >> 2008-08-07 19:56:32 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-1: > >> cleaning up state in transport object 0x961f350 > >> 2008-08-07 19:56:32 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: > >> forced unwinding frame type(1) op(34) reply=@0x962f690 > >> 2008-08-07 19:56:32 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-1: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:56:32 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-1 returned errno 107 > >> 2008-08-07 19:56:32 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-1: > >> forced unwinding frame type(1) op(34) reply=@0x962f690 > >> 2008-08-07 19:56:32 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-1: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:56:32 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-1 returned errno 107 > >> 2008-08-07 19:56:35 W [client-protocol.c:205:call_bail] > >> client-stripe-4: activating bail-out. pending frames = 2. last > sent = > >> 2008-08-07 19:55:45. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:56:35 C [client-protocol.c:212:call_bail] > >> client-stripe-4: bailing transport > >> 2008-08-07 19:56:35 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-4: > >> cleaning up state in transport object 0x9629f20 > >> 2008-08-07 19:56:35 W [client-protocol.c:205:call_bail] > >> client-stripe-3: activating bail-out. pending frames = 2. last > sent = > >> 2008-08-07 19:55:45. last received = 1969-12-31 19:00:00 > >> transport-timeout = 42 > >> 2008-08-07 19:56:35 C [client-protocol.c:212:call_bail] > >> client-stripe-3: bailing transport > >> 2008-08-07 19:56:35 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: > >> forced unwinding frame type(1) op(34) reply=@0x962c870 > >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-4: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:56:35 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-4 returned errno 107 > >> 2008-08-07 19:56:35 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-4: > >> forced unwinding frame type(1) op(34) reply=@0x962c870 > >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-4: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:56:35 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-4 returned errno 107 > >> 2008-08-07 19:56:35 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-3: > >> cleaning up state in transport object 0x96278c0 > >> 2008-08-07 19:56:35 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: > >> forced unwinding frame type(1) op(34) reply=@0x9629c90 > >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-3: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:56:35 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-3 returned errno 107 > >> 2008-08-07 19:56:35 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-3: > >> forced unwinding frame type(1) op(34) reply=@0x9629c90 > >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-3: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:56:35 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-3 returned errno 107 > >> 2008-08-07 19:56:35 E [fuse-bridge.c:468:fuse_entry_cbk] > >> glusterfs-fuse: 63764: (34) / => -1 (107) > >> 2008-08-07 19:56:35 E > [ib-verbs-client.c:141:ib_verbs_client_connect] > >> client-stripe-1: connection not in progress - trace: Network is > >> unreachable > >> 2008-08-07 19:56:35 W [client-protocol.c:332:client_protocol_xfer] > >> client-stripe-1: not connected at the moment to submit frame > type(1) > >> op(34) > >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-1: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:56:35 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-1 returned errno 107 > >> 2008-08-07 19:56:35 E > [ib-verbs-client.c:141:ib_verbs_client_connect] > >> client-stripe-2: connection not in progress - trace: Network is > >> unreachable > >> 2008-08-07 19:56:35 W [client-protocol.c:332:client_protocol_xfer] > >> client-stripe-2: not connected at the moment to submit frame > type(1) > >> op(34) > >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-2: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:56:35 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-2 returned errno 107 > >> 2008-08-07 19:56:35 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-3: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 19:56:35 E > [ib-verbs-client.c:141:ib_verbs_client_connect] > >> client-stripe-4: connection not in progress - trace: Network is > >> unreachable > >> 2008-08-07 19:56:35 W [client-protocol.c:332:client_protocol_xfer] > >> client-stripe-4: not connected at the moment to submit frame > type(1) > >> op(34) > >> 2008-08-07 19:56:35 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-4: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:56:35 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-4 returned errno 107 > >> 2008-08-07 19:56:45 E > [ib-verbs-client.c:141:ib_verbs_client_connect] > >> client-stripe-4: connection not in progress - trace: Network is > >> unreachable > >> 2008-08-07 19:57:25 E > [ib-verbs-client.c:141:ib_verbs_client_connect] > >> client-stripe-3: connection not in progress - trace: Network is > >> unreachable > >> 2008-08-07 19:57:28 E > [ib-verbs-client.c:141:ib_verbs_client_connect] > >> client-stripe-1: connection not in progress - trace: Network is > >> unreachable > >> 2008-08-07 19:57:28 W [client-protocol.c:332:client_protocol_xfer] > >> client-stripe-1: not connected at the moment to submit frame > type(1) > >> op(34) > >> 2008-08-07 19:57:28 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-1: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:57:28 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-1 returned errno 107 > >> 2008-08-07 19:57:28 E > [ib-verbs-client.c:141:ib_verbs_client_connect] > >> client-stripe-2: connection not in progress - trace: Network is > >> unreachable > >> 2008-08-07 19:57:28 W [client-protocol.c:332:client_protocol_xfer] > >> client-stripe-2: not connected at the moment to submit frame > type(1) > >> op(34) > >> 2008-08-07 19:57:28 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-2: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:57:28 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-2 returned errno 107 > >> 2008-08-07 19:57:28 E > [ib-verbs-client.c:141:ib_verbs_client_connect] > >> client-stripe-3: connection not in progress - trace: Network is > >> unreachable > >> 2008-08-07 19:57:28 W [client-protocol.c:332:client_protocol_xfer] > >> client-stripe-3: not connected at the moment to submit frame > type(1) > >> op(34) > >> 2008-08-07 19:57:28 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-3: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:57:28 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-3 returned errno 107 > >> 2008-08-07 19:57:28 E > [ib-verbs-client.c:141:ib_verbs_client_connect] > >> client-stripe-4: connection not in progress - trace: Network is > >> unreachable > >> 2008-08-07 19:57:28 W [client-protocol.c:332:client_protocol_xfer] > >> client-stripe-4: not connected at the moment to submit frame > type(1) > >> op(34) > >> 2008-08-07 19:57:28 E [client-protocol.c:4423:client_lookup_cbk] > >> client-stripe-4: no proper reply from server, returning ENOTCONN > >> 2008-08-07 19:57:28 W > >> [stripe.c:412:stripe_stack_unwind_inode_lookup_cbk] stripe: > >> client-stripe-4 returned errno 107 > >> 2008-08-07 19:57:28 E [fuse-bridge.c:468:fuse_entry_cbk] > >> glusterfs-fuse: 63765: (34) /iozone.DUMMY.14 => -1 (107) > >> 2008-08-07 19:57:29 E > [ib-verbs-client.c:141:ib_verbs_client_connect] > >> client-stripe-2: connection not in progress - trace: Network is > >> unreachable > >> 2008-08-07 19:57:30 W [client-protocol.c:1296:client_close] > >> client-stripe-2: no valid fd found, returning > >> 2008-08-07 19:57:30 W [client-protocol.c:1296:client_close] > >> client-stripe-3: no valid fd found, returning > >> 2008-08-07 19:57:30 W [client-protocol.c:1296:client_close] > >> client-stripe-4: no valid fd found, returning > >> 2008-08-07 19:57:30 W [client-protocol.c:1296:client_close] > >> client-stripe-1: no valid fd found, returning > >> 2008-08-07 19:57:33 E > [ib-verbs-client.c:141:ib_verbs_client_connect] > >> client-stripe-1: connection not in progress - trace: Network is > >> unreachable > >> 2008-08-07 20:00:04 W [client-protocol.c:280:client_protocol_xfer] > >> trans: attempting to pipeline request type(2) op(4) with handshake > >> 2008-08-07 20:00:04 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-2: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 20:00:04 W [fuse-bridge.c:398:fuse_entry_cbk] > >> glusterfs-fuse: 2: (34) / => 1 Rehashing 0/0 > >> 2008-08-07 23:31:29 W [client-protocol.c:280:client_protocol_xfer] > >> trans: attempting to pipeline request type(2) op(4) with handshake > >> 2008-08-07 23:31:29 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-3: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 23:31:29 W [fuse-bridge.c:398:fuse_entry_cbk] > >> glusterfs-fuse: 2: (34) / => 1 Rehashing 0/0 > >> 2008-08-07 23:59:06 W [client-protocol.c:205:call_bail] > >> client-stripe-2: activating bail-out. pending frames = 1. last > sent = > >> 2008-08-07 23:58:23. last received = 2008-08-07 23:58:17 > >> transport-timeout = 42 > >> 2008-08-07 23:59:06 C [client-protocol.c:212:call_bail] > >> client-stripe-2: bailing transport > >> 2008-08-07 23:59:06 W > >> [client-protocol.c:4777:client_protocol_cleanup] client-stripe-2: > >> cleaning up state in transport object 0x87eb260 > >> 2008-08-07 23:59:06 E > >> [client-protocol.c:4827:client_protocol_cleanup] client-stripe-2: > >> forced unwinding frame type(1) op(14) reply=@0x87f3d10 > >> 2008-08-07 23:59:06 E [client-protocol.c:3247:client_write_cbk] > >> client-stripe-2: no proper reply from server, returning ENOTCONN > >> 2008-08-07 23:59:06 W [stripe.c:3070:stripe_writev_cbk] stripe: > >> client-stripe-2 returned errno 107 > >> 2008-08-07 23:59:06 E [fuse-bridge.c:1645:fuse_writev_cbk] > >> glusterfs-fuse: 128670: WRITE => -1 (107) > >> 2008-08-07 23:59:06 W [client-protocol.c:280:client_protocol_xfer] > >> client-stripe-2: attempting to pipeline request type(1) op(34) with > >> handshake > >> 2008-08-07 23:59:28 E [client-protocol.c:1238:client_flush] > >> client-stripe-2: : returning EBADFD > >> 2008-08-07 23:59:28 W [stripe.c:198:stripe_stack_unwind_cbk] > stripe: > >> client-stripe-2 returned errno 77 > >> 2008-08-07 23:59:28 W [client-protocol.c:1296:client_close] > >> client-stripe-2: no valid fd found, returning > >> 2008-08-08 00:51:00 W [client-protocol.c:1296:client_close] > >> client-stripe-2: no valid fd found, returning > >> > >> > >> -Peter > >> > > > -- > Peter Pelekis 5444 Napa Street > Western Scientific San Diego CA, 92110 > Senior Systems Engineer www.wsm.com <http://www.wsm.com> > pete.pelekis at wsm.com <mailto:pete.pelekis at wsm.com> Fax > 619-220-6590 > Phone 619-220-6580 x212 Toll Free 800-443-6699 > GSA# GS-35F-5009H > > Visit us at: > > "I'm a great believer in luck, and I find the harder I work the > more I have of it." > -Thomas Jefferson > > "If everything seems under control, you're not going fast enough." > -Mario Andretti > > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org> > http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users > > > > > -- > Amar Tumballi > Gluster/GlusterFS Hacker > [bulde on #gluster/irc.gnu.org <http://irc.gnu.org>] > http://www.zresearch.com - Commoditizing Super Storage! -- Peter Pelekis 5444 Napa Street Western Scientific San Diego CA, 92110 Senior Systems Engineer www.wsm.com pete.pelekis at wsm.com Fax 619-220-6590 Phone 619-220-6580 x212 Toll Free 800-443-6699 GSA# GS-35F-5009H Visit us at: "I'm a great believer in luck, and I find the harder I work the more I have of it." -Thomas Jefferson "If everything seems under control, you're not going fast enough." -Mario Andretti
