Hi! 2008/8/11 Anand Avati <avati@xxxxxxxxxxxxx>: > can you please repost the logs "with" the timestamps? > Here there are: ----------- 2008-07-28 16:04:47 E [client-protocol.c:4430:client_lookup_cbk] c54: no proper reply from server, returning ENOTCONN 2008-07-28 16:05:38 W [client-protocol.c:205:call_bail] c-ns: activating bail-out. pending frames = 11. last sent = 2008-07-28 16:04:47. last received = 2008-07-28 16:03:17 transport-timeout = 42 2008-07-28 16:05:38 C [client-protocol.c:212:call_bail] c-ns: bailing transport 2008-07-28 16:05:38 W [client-protocol.c:4784:client_protocol_cleanup] c-ns: cleaning up state in transport object 0x64d310 2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(36) reply=@0xc165b0 2008-07-28 16:05:38 E [client-protocol.c:4215:client_setdents_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(36) reply=@0xc165b0 2008-07-28 16:05:38 E [client-protocol.c:4215:client_setdents_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(36) reply=@0xc165b0 2008-07-28 16:05:38 E [client-protocol.c:4215:client_setdents_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(36) reply=@0xc165b0 2008-07-28 16:05:38 E [client-protocol.c:4215:client_setdents_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(36) reply=@0xc165b0 2008-07-28 16:05:38 E [client-protocol.c:4215:client_setdents_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(23) reply=@0xc165b0 2008-07-28 16:05:38 E [client-protocol.c:3310:client_getdents_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:05:38 W [client-protocol.c:1711:client_closedir] c51: no proper fd found, returning 2008-07-28 16:05:38 W [client-protocol.c:1711:client_closedir] c-ns: no proper fd found, returning 2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(34) reply=@0xc165b0 2008-07-28 16:05:38 E [client-protocol.c:4430:client_lookup_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:05:38 E [unify.c:182:unify_lookup_cbk] bricks: c-ns returned 107 2008-07-28 16:05:38 E [fuse-bridge.c:468:fuse_entry_cbk] glusterfs-fuse: 1976200: (34) /nova => -1 (107) 2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer] c51: attempting to pipeline request type(1) op(34) with handshake 2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer] c37: attempting to pipeline request type(1) op(34) with handshake 2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer] c54: attempting to pipeline request type(1) op(34) with handshake 2008-07-28 16:05:38 E [client-protocol.c:325:client_protocol_xfer] c-ns: transport_submit failed 2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(34) reply=@0xc165b0 2008-07-28 16:05:38 E [client-protocol.c:4430:client_lookup_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:05:38 E [fuse-bridge.c:468:fuse_entry_cbk] glusterfs-fuse: 1976201: (34) / => -1 (2) 2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(0) reply=@0xc165b0 2008-07-28 16:05:38 E [client-protocol.c:2688:client_stat_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:05:38 E [unify.c:118:unify_buf_cbk] bricks: c-ns returned 107 2008-07-28 16:05:38 E [fuse-bridge.c:589:fuse_attr_cbk] glusterfs-fuse: 1976202: (0) /nova/nio1/new_free/m085am200_low_turb => -1 (107) 2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(2) op(9) reply=@0xc165b0 2008-07-28 16:05:38 E [client-protocol.c:4613:client_checksum_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(2) op(9) reply=@0xc165b0 2008-07-28 16:05:38 E [client-protocol.c:4613:client_checksum_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer] c54: attempting to pipeline request type(1) op(35) with handshake 2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer] c-ns: attempting to pipeline request type(1) op(35) with handshake 2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer] c37: attempting to pipeline request type(1) op(35) with handshake 2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer] c51: attempting to pipeline request type(1) op(35) with handshake 2008-07-28 16:05:38 E [unify.c:325:unify_lookup] bricks: returning ESTALE for /nova/nio1 [translator generation (27) inode generation (25)] 2008-07-28 16:05:38 E [fuse-bridge.c:468:fuse_entry_cbk] glusterfs-fuse: 1976204: (34) /nova/nio1 => -1 (116) 2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer] c51: attempting to pipeline request type(1) op(34) with handshake 2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer] c37: attempting to pipeline request type(1) op(34) with handshake 2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer] c54: attempting to pipeline request type(1) op(34) with handshake 2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer] c-ns: attempting to pipeline request type(1) op(34) with handshake 2008-07-28 16:06:24 W [client-protocol.c:205:call_bail] c37: activating bail-out. pending frames = 3. last sent = 2008-07-28 16:05:38. last received = 1970-01-01 03:00:00 transport-timeout = 42 ----------- 2008-07-28 16:06:24 E [client-protocol.c:4430:client_lookup_cbk] c37: no proper reply from server, returning ENOTCONN 2008-07-28 16:06:28 W [client-protocol.c:205:call_bail] c54: activating bail-out. pending frames = 3. last sent = 2008-07-28 16:05:38. last received = 1970-01-01 03:00:00 transport-timeout = 42 2008-07-28 16:06:28 C [client-protocol.c:212:call_bail] c54: bailing transport 2008-07-28 16:06:28 W [client-protocol.c:205:call_bail] c-ns: activating bail-out. pending frames = 3. last sent = 2008-07-28 16:05:38. last received = 1970-01-01 03:00:00 transport-timeout = 42 2008-07-28 16:06:28 C [client-protocol.c:212:call_bail] c-ns: bailing transport ----------- 2008-07-28 16:06:28 W [client-protocol.c:332:client_protocol_xfer] c-ns: not connected at the moment to submit frame type(1) op(35) 2008-07-28 16:06:28 W [client-protocol.c:332:client_protocol_xfer] c-ns: not connected at the moment to submit frame type(1) op(35) 2008-07-28 16:06:28 W [client-protocol.c:332:client_protocol_xfer] c-ns: not connected at the moment to submit frame type(1) op(35) And hundreds of the same lines during this and following second. ----------- 2008-07-28 16:10:36 E [client-protocol.c:4430:client_lookup_cbk] c54: no proper reply from server, returning ENOTCONN 2008-07-28 16:11:21 W [client-protocol.c:205:call_bail] c-ns: activating bail-out. pending frames = 5. last sent = 2008-07-28 16:10:36. last received = 2008-07-28 16:10:36 transport-timeout = 42 2008-07-28 16:11:21 C [client-protocol.c:212:call_bail] c-ns: bailing transport 2008-07-28 16:11:21 W [client-protocol.c:4784:client_protocol_cleanup] c-ns: cleaning up state in transport object 0x64d310 2008-07-28 16:11:21 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(36) reply=@0x2aaab0048720 2008-07-28 16:11:21 E [client-protocol.c:4215:client_setdents_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:11:21 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(36) reply=@0x2aaab0048720 2008-07-28 16:11:21 E [client-protocol.c:4215:client_setdents_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:11:21 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(23) reply=@0x2aaab0048720 2008-07-28 16:11:21 E [client-protocol.c:3310:client_getdents_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:11:21 W [client-protocol.c:1711:client_closedir] c-ns: no proper fd found, returning 2008-07-28 16:11:21 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(34) reply=@0x2aaab0048720 2008-07-28 16:11:21 E [client-protocol.c:4430:client_lookup_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:11:21 E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced unwinding frame type(1) op(22) reply=@0x2aaab0048720 2008-07-28 16:11:21 E [client-protocol.c:3767:client_opendir_cbk] c-ns: no proper reply from server, returning ENOTCONN 2008-07-28 16:12:08 W [client-protocol.c:205:call_bail] c51: activating bail-out. pending frames = 3. last sent = 2008-07-28 16:11:21. last received = 2008-07-28 16:10:36 transport-timeout = 42 2008-07-28 16:12:08 C [client-protocol.c:212:call_bail] c51: bailing transport 2008-07-28 16:12:08 W [client-protocol.c:4784:client_protocol_cleanup] c51: cleaning up state in transport object 0x614600 2008-07-28 16:12:08 E [client-protocol.c:4834:client_protocol_cleanup] c51: forced unwinding frame type(1) op(34) reply=@0x2aaab00180c0 2008-07-28 16:12:08 E [client-protocol.c:4430:client_lookup_cbk] c51: no proper reply from server, returning ENOTCONN 2008-07-28 16:12:08 E [fuse-bridge.c:468:fuse_entry_cbk] glusterfs-fuse: 1978748: (34) /nova => -1 (2) 2008-07-28 16:12:08 E [client-protocol.c:4834:client_protocol_cleanup] c51: forced unwinding frame type(1) op(22) reply=@0x2aaab00180c0 ---------- 2008-08-04 13:10:09 E [unify.c:118:unify_buf_cbk] bricks: c41 returned 77 2008-08-04 13:10:09 E [client-protocol.c:1884:client_fstat] c-ns: : returning EBADFD 2008-08-04 13:10:09 E [unify.c:118:unify_buf_cbk] bricks: c-ns returned 77 2008-08-04 13:10:09 E [fuse-bridge.c:589:fuse_attr_cbk] glusterfs-fuse: 9731: (29) /ansys11/ansys_inc/shared_files/licensing/linem64t/ansyslmd => -1 (77) ---------- Hope this helps. I can upload full logs somewhere. Regards! Andrey > > 2008/8/11 NovA <av.nova@xxxxxxxxx> >> >> Hi! >> >> I'm using glusterfs-1.3.9tla790 unify (without AFR) and also >> periodically has troubles with NS. Sometimes, when many files are >> copied to unify volume, the operation is stalled. And the client log >> is flooded by messages like: >> W [client-protocol.c:332:client_protocol_xfer] c-ns: not connected at >> the moment to submit frame type(1) op(35) >> The glusterfs server not really crash, but eat 100% CPU. Restarting it >> restores the normal work. >> >> There are also many other messages concerning namespace, like >> ------ >> W [client-protocol.c:1711:client_closedir] c-ns: no proper fd found, >> returning >> E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced >> unwinding frame type(1) op(34) reply=@0xc165b0 >> E [client-protocol.c:4430:client_lookup_cbk] c-ns: no proper reply >> from server, returning ENOTCONN >> E [unify.c:182:unify_lookup_cbk] bricks: c-ns returned 107 >> ------ >> W [client-protocol.c:205:call_bail] c-ns: activating bail-out. pending >> frames = 3. last sent = 2008-07-28 16:05:38. last received = >> 1970-01-01 03:00:00 transport-timeout = 42 >> C [client-protocol.c:212:call_bail] c-ns: bailing transport >> ------ >> Note the strange last received date... >> ----- >> W [client-protocol.c:280:client_protocol_xfer] c-ns: attempting to >> pipeline request type(1) op(34) with handshake >> ----- >> W [client-protocol.c:4784:client_protocol_cleanup] c-ns: cleaning up >> state in transport object 0x64 >> E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced >> unwinding frame type(1) op(34) reply=@0x66c140 >> E [client-protocol.c:4430:client_lookup_cbk] c-ns: no proper reply >> from server, returning ENOTCONN >> ---- >> [client-protocol.c:4784:client_protocol_cleanup] c-ns: cleaning up >> state in transport object 0x64d310 >> [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced >> unwinding frame type(1) op(36) reply=@0x2aaab0048720 >> [client-protocol.c:4215:client_setdents_cbk] c-ns: no proper reply >> from server, returning ENOTCONN >> [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced >> unwinding frame type(1) op(36) reply=@0x2aaab0048720 >> [client-protocol.c:4215:client_setdents_cbk] c-ns: no proper reply >> from server, returning ENOTCONN >> [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced >> unwinding frame type(1) op(23) reply=@0x2aaab0048720 >> [client-protocol.c:3310:client_getdents_cbk] c-ns: no proper reply >> from server, returning ENOTCONN >> [client-protocol.c:1711:client_closedir] c-ns: no proper fd found, >> returning >> [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced >> unwinding frame type(1) op(34) reply=@0x2aaab0048720 >> [client-protocol.c:4430:client_lookup_cbk] c-ns: no proper reply from >> server, returning ENOTCONN >> [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced >> unwinding frame type(1) op(22) reply=@0x2aaab0048720 >> [client-protocol.c:3767:client_opendir_cbk] c-ns: no proper reply from >> server, returning ENOTCONN >> ---- >> E [client-protocol.c:1884:client_fstat] c-ns: : returning EBADFD >> E [unify.c:118:unify_buf_cbk] bricks: c-ns returned 77 >> ---- >> >> All these are from different places of the log (rather huge now), and >> probably not connected to each other. It's just to show different >> types of warnings and errors I've really saw. >> >> The name-space in my case is just a folder on common ext3 volume. It's >> for testing purposes, I'm going to use separate reiserfs volume for it >> later. But probably it's somehow connected with NS problems which I'm >> seeing now. >>
