Hi Avati,
today i found some difference in date & time on the three machines. I
correct the issue, so that all 3 machines have the same date and time.
It seems to work now, I will make some further tests and if I have
problems I will send you more logs.
My Setup uses 3 machines. 2 Servers an one seperate client. Server and
client are not the same machine.
Thanks
Urban
Anand Avati wrote:
Urban,
is it possible to get the logs from a little before what you pasted?
also, was the machine you brought down both server and a client ?
avati
2007/7/20, Urban Loesch < ul@xxxxxxxx <mailto:ul@xxxxxxxx>>:
Hi Avati,
still the Problem when I shut down the second server.
Here the Log
2007-07-20 18:51:15 C [tcp.c:81:tcp_disconnect] server: connection
disconnected
2007-07-20 18:51:15 E [client-protocol.c:336:client_protocol_xfer]
data1-gluster2-ds: transport_submit failed
2007-07-20 18:51:15 W [client-protocol.c:4196:client_protocol_cleanup]
data1-gluster2-ds: cleaning up state in transport object 0x8052bd8
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
data1-gluster2-ds: forced unwinding frame type(0) op(14)
2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
(path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
2007-07-20 18:51:15 C [common-utils.c:210:gf_print_trace]
debug-backtrace: Got signal (11), printing backtrace
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace: /usr/lib/libglusterfs.so.0(gf_print_trace+0x2d)
[0xb7edc32d]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace: /lib/i686/cmov/libc.so.6 [0xb7d98208]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace: /usr/lib/libglusterfs.so.0(dict_ref+0x29)
[0xb7ed5989]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace:
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so [0xb752b018]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace:
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so [0xb752c185]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace:
/usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so
[0xb7538b1f]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace:
/usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so
[0xb753d1d5]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace: /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so
[0xb754bef3]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace:
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so [0xb756203f]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace:
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so(notify+0x46b)
[0xb756072b]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace: /usr/lib/libglusterfs.so.0(transport_notify+0x37)
[0xb7edd837]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace: /usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xd9)
[0xb7ede359]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace: /usr/lib/libglusterfs.so.0(poll_iteration+0x1d)
[0xb7edd90d]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace: [glusterfsd] [0x80493ee]
2007-07-20 18:51:15 C [common-utils.c :212:gf_print_trace]
debug-backtrace: /lib/i686/cmov/libc.so.6(__libc_start_main+0xe0)
[0xb7d84030]
2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
debug-backtrace: [glusterfsd] [0x8048d81]
And the backtrace:
gluster1:/# gdb glusterfsd -c core.15573
GNU gdb 6.4.90-debian
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License,
and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i486-linux-gnu"...(no debugging
symbols found)
Using host libthread_db library "/lib/i686/cmov/libthread_db.so.1".
Reading symbols from /usr/lib/libglusterfs.so.0...(no debugging
symbols
found)...done.
Loaded symbols for /usr/lib/libglusterfs.so.0
Reading symbols from /lib/i686/cmov/libdl.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/i686/cmov/libdl.so.2
Reading symbols from /lib/i686/cmov/libpthread.so.0...(no debugging
symbols found)...done.
Loaded symbols for /lib/i686/cmov/libpthread.so.0
Reading symbols from /lib/i686/cmov/libc.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib/i686/cmov/libc.so.6
Reading symbols from /lib/ld- linux.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from
/usr/lib/glusterfs/1.3.tla-322/xlator/storage/posix.so...
(no debugging symbols found)...done.
Loaded symbols for
/usr/lib/glusterfs/1.3.tla-322/xlator/storage/posix.so
Reading symbols from
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so...(no
debugging
symbols found)...done.
Loaded symbols for
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so
Reading symbols from
/usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so...(no debugging
symbols found)...done.
Loaded symbols for
/usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so
Reading symbols from
/usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so...(no debugging
symbols found)...done.
Loaded symbols for
/usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so
Reading symbols from
/usr/lib/glusterfs/1.3.tla-322/xlator/performance/io- threads.so...(no
debugging symbols found)...done.
Loaded symbols for
/usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so
Reading symbols from
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so...
(no debugging symbols found)...done.
Loaded symbols for
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so
Reading symbols from
/usr/lib/glusterfs/1.3.tla-322/transport/tcp/client.so...(no debugging
symbols found)...done.
Loaded symbols for
/usr/lib/glusterfs/1.3.tla-322/transport/tcp/client.so
Reading symbols from
/usr/lib/glusterfs/1.3.tla-322/scheduler/rr.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/scheduler/rr.so
Reading symbols from
/usr/lib/glusterfs/1.3.tla-322/transport/tcp/server.so...(no debugging
symbols found)...done.
Loaded symbols for
/usr/lib/glusterfs/1.3.tla-322/transport/tcp/server.so
Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libgcc_s.so.1
Core was generated by
`[glusterfsd]
'.
Program terminated with signal 11, Segmentation fault.
#0 0xb7ebc430 in pthread_mutex_lock () from
/lib/i686/cmov/libpthread.so.0
(gdb) backtrace
#0 0xb7ebc430 in pthread_mutex_lock () from
/lib/i686/cmov/libpthread.so.0
#1 0xb7ed5989 in dict_ref () from /usr/lib/libglusterfs.so.0
#2 0xb752b018 in fini () from
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so
#3 0xb752c185 in mop_lock () from
/usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so
#4 0xb7538b1f in ?? () from
/usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so
#5 0x081f1924 in ?? ()
#6 0x081e2a08 in ?? ()
#7 0x08051970 in ?? ()
#8 0x00020000 in ?? ()
#9 0x00000022 in ?? ()
#10 0xbfadbdf0 in ?? ()
#11 0xbfadbd48 in ?? ()
#12 0xb753d1d5 in ?? () from
/usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so
#13 0x081e2a08 in ?? ()
#14 0x081ec2c8 in ?? ()
#15 0x080518f8 in ?? ()
#16 0x00020000 in ?? ()
#17 0x00000022 in ?? ()
#18 0xbfadbdf0 in ?? ()
#19 0xbfadbd88 in ?? ()
#20 0xb754bef3 in fini () from
/usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so
Previous frame inner to this frame (corrupt stack?)
Thanks and regards
Urban
--
Anand V. Avati
