Urban, is it possible to get the logs from a little before what you pasted? also, was the machine you brought down both server and a client ? avati 2007/7/20, Urban Loesch <ul@xxxxxxxx>:
Hi Avati, still the Problem when I shut down the second server. Here the Log 2007-07-20 18:51:15 C [tcp.c:81:tcp_disconnect] server: connection disconnected 2007-07-20 18:51:15 E [client-protocol.c:336:client_protocol_xfer] data1-gluster2-ds: transport_submit failed 2007-07-20 18:51:15 W [client-protocol.c:4196:client_protocol_cleanup] data1-gluster2-ds: cleaning up state in transport object 0x8052bd8 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] data1-gluster2-ds: forced unwinding frame type(0) op(14) 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] data1-gluster2-ds: forced unwinding frame type(0) op(14) 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] data1-gluster2-ds: forced unwinding frame type(0) op(14) 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] data1-gluster2-ds: forced unwinding frame type(0) op(14) 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] data1-gluster2-ds: forced unwinding frame type(0) op(14) 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] data1-gluster2-ds: forced unwinding frame type(0) op(14) 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] data1-gluster2-ds: forced unwinding frame type(0) op(14) 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] data1-gluster2-ds: forced unwinding frame type(0) op(14) 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup] data1-gluster2-ds: forced unwinding frame type(0) op(14) 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr: (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107 2007-07-20 18:51:15 C [common-utils.c:210:gf_print_trace] debug-backtrace: Got signal (11), printing backtrace 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(gf_print_trace+0x2d) [0xb7edc32d] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /lib/i686/cmov/libc.so.6 [0xb7d98208] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(dict_ref+0x29) [0xb7ed5989] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so [0xb752b018] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so [0xb752c185] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so[0xb7538b1f] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so [0xb753d1d5] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so [0xb754bef3] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so [0xb756203f] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so(notify+0x46b) [0xb756072b] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(transport_notify+0x37) [0xb7edd837] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xd9) [0xb7ede359] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(poll_iteration+0x1d) [0xb7edd90d] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: [glusterfsd] [0x80493ee] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: /lib/i686/cmov/libc.so.6(__libc_start_main+0xe0) [0xb7d84030] 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace] debug-backtrace: [glusterfsd] [0x8048d81] And the backtrace: gluster1:/# gdb glusterfsd -c core.15573 GNU gdb 6.4.90-debian Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i486-linux-gnu"...(no debugging symbols found) Using host libthread_db library "/lib/i686/cmov/libthread_db.so.1". Reading symbols from /usr/lib/libglusterfs.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libglusterfs.so.0 Reading symbols from /lib/i686/cmov/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/i686/cmov/libdl.so.2 Reading symbols from /lib/i686/cmov/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/i686/cmov/libpthread.so.0 Reading symbols from /lib/i686/cmov/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/i686/cmov/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /usr/lib/glusterfs/1.3.tla-322/xlator/storage/posix.so... (no debugging symbols found)...done. Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/storage/posix.so Reading symbols from /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so Reading symbols from /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so Reading symbols from /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so Reading symbols from /usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so Reading symbols from /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so... (no debugging symbols found)...done. Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so Reading symbols from /usr/lib/glusterfs/1.3.tla-322/transport/tcp/client.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/transport/tcp/client.so Reading symbols from /usr/lib/glusterfs/1.3.tla-322/scheduler/rr.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/scheduler/rr.so Reading symbols from /usr/lib/glusterfs/1.3.tla-322/transport/tcp/server.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/transport/tcp/server.so Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libgcc_s.so.1 Core was generated by `[glusterfsd] '. Program terminated with signal 11, Segmentation fault. #0 0xb7ebc430 in pthread_mutex_lock () from /lib/i686/cmov/libpthread.so.0 (gdb) backtrace #0 0xb7ebc430 in pthread_mutex_lock () from /lib/i686/cmov/libpthread.so.0 #1 0xb7ed5989 in dict_ref () from /usr/lib/libglusterfs.so.0 #2 0xb752b018 in fini () from /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so #3 0xb752c185 in mop_lock () from /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so #4 0xb7538b1f in ?? () from /usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so #5 0x081f1924 in ?? () #6 0x081e2a08 in ?? () #7 0x08051970 in ?? () #8 0x00020000 in ?? () #9 0x00000022 in ?? () #10 0xbfadbdf0 in ?? () #11 0xbfadbd48 in ?? () #12 0xb753d1d5 in ?? () from /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so #13 0x081e2a08 in ?? () #14 0x081ec2c8 in ?? () #15 0x080518f8 in ?? () #16 0x00020000 in ?? () #17 0x00000022 in ?? () #18 0xbfadbdf0 in ?? () #19 0xbfadbd88 in ?? () #20 0xb754bef3 in fini () from /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so Previous frame inner to this frame (corrupt stack?) Thanks and regards Urban
-- Anand V. Avati
