Hi Pooya,
let us know your setup.. ie, what application you were running and howmany
clients.
-amar
On 7/12/07, Pooya Woodcock <pooya@xxxxxxxxxxxxxxx> wrote:
Also on another node, I got this segfault. This one was first then I
got the other one on node4 that I emailed eearlier.
[root@node02 ~]# gdb glusterfs -c /core.13358
GNU gdb Red Hat Linux (6.3.0.0-1.132.EL4rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host
libthread_db library "/lib64/tls/libthread_db.so.1".
Core was generated by `
[glusterfs]
'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/local/lib/libglusterfs.so.0...done.
Loaded symbols for /usr/local/lib/libglusterfs.so.0
Reading symbols from /usr/local/lib/libibverbs.so.1...done.
Loaded symbols for /usr/local/lib/libibverbs.so.1
Reading symbols from /usr/local/lib/libfuse.so.2...done.
Loaded symbols for /usr/local/lib/libfuse.so.2
Reading symbols from /lib64/libdl.so.2...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/tls/libpthread.so.0...done.
Loaded symbols for /lib64/tls/libpthread.so.0
Reading symbols from /lib64/tls/libc.so.6...done.
Loaded symbols for /lib64/tls/libc.so.6
Reading symbols from /usr/lib64/libsysfs.so.1...done.
Loaded symbols for /usr/lib64/libsysfs.so.1
Reading symbols from /lib64/tls/librt.so.1...done.
Loaded symbols for /lib64/tls/librt.so.1
Reading symbols from /lib64/libselinux.so.1...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
protocol/client.so...done.
Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
protocol/client.so
Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
cluster/afr.so...done.
Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
cluster/afr.so
Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
cluster/unify.so...done.
Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
cluster/unify.so
Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
performance/io-threads.so...done.
Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
performance/io-threads.so
Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
performance/read-ahead.so...done.
Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
performance/read-ahead.so
Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/transport/
tcp/client.so...done.
Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/transport/
tcp/client.so
Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/scheduler/
rr.so...done.
Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/scheduler/rr.so
Reading symbols from /lib64/libgcc_s.so.1...done.
Loaded symbols for /lib64/libgcc_s.so.1
#0 unify_open (frame=0x2a960d3ed0, this=0x514b70, loc=0x2a960cd178,
flags=32770) at unify.c:956
956 loc_t tmp_loc = {
(gdb) bt
#0 unify_open (frame=0x2a960d3ed0, this=0x514b70, loc=0x2a960cd178,
flags=32770) at unify.c:956
#1 0x0000002a95bc8121 in iot_open (frame=0x2a960d2190,
this=0x515700, loc=0x2a960cd178, flags=32770) at io-threads.c:88
#2 0x0000002a95ccc2a7 in ra_open (frame=0x2a960d1800, this=0x5160a0,
loc=0x2a960cd178, flags=32770) at read-ahead.c:207
#3 0x0000000000405898 in fuse_open (req=Variable "req" is not
available.
) at fuse-bridge.c:1017
#4 0x0000002a95783ffe in do_open (req=Variable "req" is not available.
) at fuse_lowlevel.c:586
#5 0x0000000000407b52 in fuse_transport_notify (xl=Variable "xl" is
not available.
) at fuse-bridge.c:1937
#6 0x0000002a95563e86 in sys_epoll_iteration (ctx=Variable "ctx" is
not available.
) at epoll.c:54
#7 0x0000000000402f4e in main (argc=7, argv=0x7fbffff9d8) at
glusterfs.c:348
On Jul 11, 2007, at 9:58 AM, Pooya Woodcock wrote:
> Hi Avati,
> Confirmed fixed. Thanks, running great so far.
> Pooya
>
> On Jul 10, 2007, at 11:50 PM, Anand Avati wrote:
>
>> Pooya,
>> this bug is fixed in patch 295. can you please confirm if some
>> operations were performed when one (or more) server was down to
>> reproduce this bug?
>>
>> thanks,
>> avati
>>
>> 2007/7/11, Pooya Woodcock < pooya@xxxxxxxxxxxxxxx>:This is on
>> patch 293 in 2.5 mainline.
>> -Pooya
>>
>> On Jul 10, 2007, at 7:40 PM, Amar S. Tumballi wrote:
>>
>> > Hi Pooya,
>> > Thanks for the bt, but can you tell us which version (or patch in
>> > tla) are you using?
>> >
>> > -amar
>> >
>> > On 7/11/07, Pooya Woodcock < pooya@xxxxxxxxxxxxxxx> wrote:Here is
>> > gdb output on the core..
>> >
>> > [root@node02 ~]# gdb glusterfs -c /core.6612
>> > GNU gdb Red Hat Linux (6.3.0.0-1.132.EL4rh)
>> > Copyright 2004 Free Software Foundation, Inc.
>> > GDB is free software, covered by the GNU General Public License,
>> and
>> > you are
>> > welcome to change it and/or distribute copies of it under certain
>> > conditions.
>> > Type "show copying" to see the conditions.
>> > There is absolutely no warranty for GDB. Type "show warranty" for
>> > details.
>> > This GDB was configured as "x86_64-redhat-linux-gnu"...Using host
>> > libthread_db library "/lib64/tls/libthread_db.so.1".
>> >
>> > Core was generated by `
>> > [glusterfs]
>> > '.
>> > Program terminated with signal 11, Segmentation fault.
>> > Reading symbols from /usr/local/lib/libglusterfs.so.0...done.
>> > Loaded symbols for /usr/local/lib/libglusterfs.so.0
>> > Reading symbols from /usr/local/lib/libibverbs.so.1...done.
>> > Loaded symbols for /usr/local/lib/libibverbs.so.1
>> > Reading symbols from /usr/local/lib/libfuse.so.2...done.
>> > Loaded symbols for /usr/local/lib/libfuse.so.2
>> > Reading symbols from /lib64/libdl.so.2...done.
>> > Loaded symbols for /lib64/libdl.so.2
>> > Reading symbols from /lib64/tls/libpthread.so.0...done.
>> > Loaded symbols for /lib64/tls/libpthread.so.0
>> > Reading symbols from /lib64/tls/libc.so.6...done.
>> > Loaded symbols for /lib64/tls/libc.so.6
>> > Reading symbols from /usr/lib64/libsysfs.so.1...done.
>> > Loaded symbols for /usr/lib64/libsysfs.so.1
>> > Reading symbols from /lib64/tls/librt.so.1...done.
>> > Loaded symbols for /lib64/tls/librt.so.1
>> > Reading symbols from /lib64/libselinux.so.1...done.
>> > Loaded symbols for /lib64/libselinux.so.1
>> > Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
>> > Loaded symbols for /lib64/ld- linux-x86-64.so.2
>> > Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
>> > protocol/client.so...done.
>> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
>> > protocol/client.so
>> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/
>> > cluster/afr.so...done.
>> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/
>> > cluster/afr.so
>> > Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
>> > cluster/unify.so...done.
>> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
>> > cluster/unify.so
>> > Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
>> > performance/io-threads.so...done.
>> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/
>> > performance/io- threads.so
>> > Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
>> > performance/read-ahead.so...done.
>> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
>> > performance/read- ahead.so
>> > Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/
>> transport/
>> > tcp/client.so...done.
>> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/transport/
>> > tcp/client.so
>> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
>> scheduler/
>> > rr.so...done.
>> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/scheduler/
>> > rr.so
>> > Reading symbols from /lib64/libgcc_s.so.1...done.
>> > Loaded symbols for /lib64/libgcc_s.so.1
>> > #0 inode_unref (inode=0x0) at inode.c:403
>> > 403 inode_table_t *table = inode->table;
>> >
>> > (gdb) backtrace
>> > #0 inode_unref (inode=0x0) at inode.c:403
>> > #1 0x0000002a959ad6bc in afr_forget (frame=dwarf2_read_address:
>> > Corrupted DWARF expression.
>> > ) at afr.c:468
>> > #2 0x0000002a95564a44 in __destroy_inode (inode=0x2a9619bc10) at
>> > inode.c:250
>> > #3 0x0000002a95564cfc in inode_unref (inode=0x2a9619bc10) at
>> inode.c:
>> > 413
>> > #4 0x0000002a95abc289 in unify_forget (frame=Variable "frame"
>> is not
>> > available.
>> > ) at unify.c:407
>> > #5 0x0000002a95564a44 in __destroy_inode (inode=0x2a9619b8b0) at
>> > inode.c:250
>> > #6 0x0000002a95564cfc in inode_unref (inode=0x2a9619b8b0) at
>> inode.c:
>> > 413
>> > #7 0x0000000000407370 in fuse_forget_notify (frame=Variable
>> "frame"
>> > is not available.
>> > ) at fuse-bridge.c:1652
>> > #8 0x0000002a95564a44 in __destroy_inode (inode=0x2a9608f0c0) at
>> > inode.c:250
>> > #9 0x0000002a95564cfc in inode_unref (inode=0x2a9608f0c0) at
>> inode.c:
>> > 413
>> > #10 0x0000000000403993 in fuse_forget (req=0x2a9607de50,
>> ino=Variable
>> > "ino" is not available.
>> > ) at fuse-bridge.c:363
>> > #11 0x0000000000407b52 in fuse_transport_notify (xl=Variable
>> "xl" is
>> > not available.
>> > ) at fuse-bridge.c:1937
>> > #12 0x0000002a95563e86 in sys_epoll_iteration (ctx=Variable
>> "ctx" is
>> > not available.
>> > ) at epoll.c:54
>> > #13 0x0000000000402f4e in main (argc=7, argv=0x7fbffffa08) at
>> > glusterfs.c:348
>> >
>> >
>> >
>> >
>> >
>> > On Jul 10, 2007, at 1:36 PM, Pooya Woodcock wrote:
>> >
>> > > 2007-07-10 13:30:21 D [ fuse-bridge.c:338:fuse_lookup] glusterfs-
>> > > fuse: LOOKUP 6832132/sess_3c1f96159c610b755a7def44f036a11b (/
>> > > php_sessions/sess_3c1f96159c610b755a
>> > > 7def44f036a11b)
>> > > 2007-07-10 13:30:21 D [inode.c:511:__create_inode] ns/inode:
>> create
>> > > inode(6832716)
>> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] ns/inode:
>> > > activating inode(6832716), lru=0/1000
>> > > 2007-07-10 13:30:21 D [inode.c:511:__create_inode] unify1/inode:
>> > > create inode(6832716)
>> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] unify1/inode:
>> > > activating inode(6832716), lru=0/1000
>> > > 2007-07-10 13:30:21 D [inode.c:511:__create_inode] brick3-afr/
>> > > inode: create inode(10535464)
>> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] brick3-afr/
>> > > inode: activating inode(10535464), lru=0/1000
>> > > 2007-07-10 13:30:21 D [inode.c:511:__create_inode] afr3/inode:
>> > > create inode(10535464)
>> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] afr3/inode:
>> > > activating inode(10535464), lru=0/1000
>> > > 2007-07-10 13:30:21 D [fuse-bridge.c:288:fuse_entry_cbk]
>> glusterfs-
>> > > fuse: ENTRY => 6832716
>> > > 2007-07-10 13:30:21 D [ inode.c:511:__create_inode] fuse/inode:
>> > > create inode(6832716)
>> > > 2007-07-10 13:30:21 D [inode.c:302:__active_inode] fuse/inode:
>> > > activating inode(6832716), lru=454/1024
>> > > 2007-07-10 13:30:21 D [ inode.c:332:__passive_inode] fuse/inode:
>> > > passivating inode(6832716), lru=455/1024
>> > > 2007-07-10 13:30:21 D [inode.c:332:__passive_inode] fuse/inode:
>> > > passivating inode(6832132), lru=456/1024
>> > > 2007-07-10 13:30:21 D [fuse-bridge.c:775:fuse_unlink] glusterfs-
>> > > fuse: UNLINK 6832132/sess_3c1f96159c610b755a7def44f036a11b
>> > > 2007-07-10 13:30:21 D [inode.c:302:__active_inode] fuse/inode:
>> > > activating inode(6832716), lru=455/1024
>> > > 2007-07-10 13:30:21 D [inode.c:302:__active_inode] fuse/inode:
>> > > activating inode(6832132), lru=454/1024
>> > > 2007-07-10 13:30:21 D [inode.c :332:__passive_inode] fuse/inode:
>> > > passivating inode(6832716), lru=455/1024
>> > > 2007-07-10 13:30:21 D [inode.c:332:__passive_inode] fuse/inode:
>> > > passivating inode(6832132), lru=456/1024
>> > > 2007-07-10 13:30:21 D [inode.c:302:__active_inode] fuse/inode:
>> > > activating inode(6832716), lru=455/1024
>> > > 2007-07-10 13:30:21 C [common-utils.c:208:gf_print_trace] debug-
>> > > backtrace: Got signal (11), printing backtrace
>> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: /usr/local/lib/libglusterfs.so.0(gf_print_trace+0x22)
>> > > [0x2a95562782]
>> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: /lib64/tls/libc.so.6 [0x37fbc2e2b0]
>> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: /usr/local/lib/libglusterfs.so.0(inode_unref+0x16)
>> > > [0x2a95564bc6]
>> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: /usr/local/lib/glusterfs/1.3.0-pre5.2 /xlator/cluster/
>> > > afr.so [0x2a959ad6bc]
>> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: /usr/local/lib/libglusterfs.so.0 [0x2a95564a44]
>> > > 2007-07-10 13:30:21 C [common-utils.c :210:gf_print_trace] debug-
>> > > backtrace: /usr/local/lib/libglusterfs.so.0(inode_unref+0x14c)
>> > > [0x2a95564cfc]
>> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/cluster/
>> > > unify.so(unify_forget+0x29) [0x2a
>> > > 95abc289]
>> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: /usr/local/lib/libglusterfs.so.0 [0x2a95564a44]
>> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: /usr/local/lib/libglusterfs.so.0(inode_unref+0x14c)
>> > > [0x2a95564cfc]
>> > > 2007-07-10 13:30:21 C [common-utils.c :210:gf_print_trace] debug-
>> > > backtrace: [glusterfs] [0x407370]
>> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: /usr/local/lib/libglusterfs.so.0 [0x2a95564a44]
>> > > 2007-07-10 13:30:21 C [common-utils.c :210:gf_print_trace] debug-
>> > > backtrace: /usr/local/lib/libglusterfs.so.0(inode_unref+0x14c)
>> > > [0x2a95564cfc]
>> > > 2007-07-10 13:30:21 C [common-utils.c :210:gf_print_trace] debug-
>> > > backtrace: [glusterfs] [0x403993]
>> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: [glusterfs] [0x407b52]
>> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: /usr/local/lib/libglusterfs.so.0(sys_epoll_iteration
>> > > +0xc6) [0x2a95563e86]
>> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: [glusterfs] [0x402f4e]
>> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: /lib64/tls/libc.so.6(__libc_start_main+0xdb)
>> > [0x37fbc1c3fb]
>> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace] debug-
>> > > backtrace: [glusterfs] [0x40290a]
>> > >
>> > >
>> > > _______________________________________________
>> > > Gluster-devel mailing list
>> > > Gluster-devel@xxxxxxxxxx
>> > > http://lists.nongnu.org/mailman/listinfo/gluster-devel
>> >
>> >
>> >
>> > _______________________________________________
>> > Gluster-devel mailing list
>> > Gluster-devel@xxxxxxxxxx
>> > http://lists.nongnu.org/mailman/listinfo/gluster-devel
>> >
>> >
>> >
>> > --
>> > Amar Tumballi
>> > http://amar.80x25.org
>> > [bulde on #gluster/irc.gnu.org]
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel@xxxxxxxxxx
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>>
>>
>> --
>> Anand V. Avati
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel@xxxxxxxxxx
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel
--
Amar Tumballi
http://amar.80x25.org
[bulde on #gluster/irc.gnu.org]
