Linux (ubuntu dapper) 2.6.15. Fuse 2.6.5. Glusterfs mainline-2.5 patch 185.
glusterfs crashing with signal 11 on 'ls'. Was working for a few hours
and now it crashes on any 'ls' of the mount. Odd thing, restarting
glusterfsd and remounting fresh and restarting rsync it seems to be
working ok (no errors). Running 6 rsyncs in parallel. As long as I dont
do an 'ls' on the mount.
Things I did (mostly in sequence) leading up to this:
* I was rsyncing for a few hours (rsync'd 75G).
* Had aborted and restarted rsync a number of times.
* Had rebooted once.
After one of these it started crashing. Dunno which cuz they were all
done about the same time:
* Tweaked network in sysctl.conf. A number of thing. Reset them back
and reboot but still crashing.
* Ran bonnie++ and aborted it.
* Added write-behind xlator to glusterfs-client.vol. Removed when it
wouldnt work with 'notify missing' error.
========================================
GDB glusterfs -c /core.5194:
Core was generated by
`[glusterfs] '.
Program terminated with signal 11, Segmentation fault.
warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libglusterfs.so.0...done.
Loaded symbols for /lib/libglusterfs.so.0
Reading symbols from /usr/local/lib/libfuse.so.2...done.
Loaded symbols for /usr/local/lib/libfuse.so.2
Reading symbols from /lib/tls/i686/cmov/libdl.so.2...done.
Loaded symbols for /lib/tls/i686/cmov/libdl.so.2
Reading symbols from /lib/tls/i686/cmov/libpthread.so.0...done.
Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0
Reading symbols from /lib/tls/i686/cmov/libc.so.6...done.
Loaded symbols for /lib/tls/i686/cmov/libc.so.6
Reading symbols from /lib/tls/i686/cmov/librt.so.1...done.
Loaded symbols for /lib/tls/i686/cmov/librt.so.1
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from
/lib/glusterfs/1.3.0-pre5/xlator/protocol/client.so...done.
Loaded symbols for /lib/glusterfs/1.3.0-pre5/xlator/protocol/client.so
Reading symbols from
/lib/glusterfs/1.3.0-pre5/xlator/cluster/unify.so...done.
Loaded symbols for /lib/glusterfs/1.3.0-pre5/xlator/cluster/unify.so
Reading symbols from
/lib/glusterfs/1.3.0-pre5/transport/tcp/client.so...done.
Loaded symbols for /lib/glusterfs/1.3.0-pre5/transport/tcp/client.so
Reading symbols from /lib/glusterfs/1.3.0-pre5/scheduler/rr.so...done.
Loaded symbols for /lib/glusterfs/1.3.0-pre5/scheduler/rr.so
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
#0 fuse_xattr_cbk (frame=0x8094e38, cookie=0x8094608, this=0x80521f0,
op_ret=0, op_errno=17, dict=0x80942b0) at fuse-bridge.c:1442
1442 ret = value_data->len;
(gdb) bt
#0 fuse_xattr_cbk (frame=0x8094e38, cookie=0x8094608, this=0x80521f0,
op_ret=0, op_errno=17, dict=0x80942b0) at fuse-bridge.c:1442
#1 0xb76564f7 in unify_getxattr_cbk (frame=0xa, cookie=0x8094190,
this=0x80570b8, op_ret=0, op_errno=0, value=0x0) at unify.c:3287
#2 0xb7663005 in client_getxattr_cbk (frame=0x8094190, args=0x8094ea8)
at client-protocol.c:3431
#3 0xb7664f4c in notify (this=0x8056e08, event=2, data=0x808e6a0) at
client-protocol.c:4126
#4 0xb7fd8982 in transport_notify (this=0x8094164, event=1) at
transport.c:152
#5 0xb7fd90fe in sys_epoll_iteration (ctx=0x0) at epoll.c:54
#6 0xb7fd8b2d in poll_iteration (ctx=0x0) at transport.c:260
#7 0x0804a329 in main (argc=3, argv=0xbfdf9c84) at glusterfs.c:341
========================================
GLUSTERFSD.LOG:
2007-06-21 14:09:47 E [protocol.c:262:gf_block_unserialize_transport]
libglusterfs/protocol: full_read of header failed: peer (127.0.0.1)
2007-06-21 14:09:47 C [tcp.c:82:tcp_disconnect] transport/tcp:
clusterfs: connection disconnected
2007-06-21 14:09:47 E [protocol.c:262:gf_block_unserialize_transport]
libglusterfs/protocol: full_read of header failed: peer (127.0.0.1)
2007-06-21 14:09:47 C [tcp.c:82:tcp_disconnect] transport/tcp:
clusterfs: connection disconnected
========================================
GLUSTERFS.LOG:
2007-06-21 14:09:47 C [common-utils.c:205:gf_print_trace]
debug-backtrace: Got signal (11), printing backtrace
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace]
debug-backtrace: /lib/libglusterfs.so.0(gf_print_trace+0x2d) [0xb7f97499]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace]
debug-backtrace: [0xffffe420]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace]
debug-backtrace: /lib/glusterfs/1.3.0-pre5/xlator/cluster/unify.so
[0xb76164f7]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace]
debug-backtrace: /lib/glusterfs/1.3.0-pre5/xlator/protocol/client.so
[0xb7623005]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace]
debug-backtrace:
/lib/glusterfs/1.3.0-pre5/xlator/protocol/client.so(notify+0x855)
[0xb7624f4c]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace]
debug-backtrace: /lib/libglusterfs.so.0(transport_notify+0x37) [0xb7f98982]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace]
debug-backtrace: /lib/libglusterfs.so.0(sys_epoll_iteration+0xd7)
[0xb7f990fe]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace]
debug-backtrace: /lib/libglusterfs.so.0(poll_iteration+0x1d) [0xb7f98b2d]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace]
debug-backtrace: [glusterfs] [0x804a329]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace]
debug-backtrace: /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xd2)
[0xb7e49ea2]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace]
debug-backtrace: [glusterfs] [0x8049c71]
========================================
glusterfs-client.vol:
volume server1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
option remote-host 127.0.0.1 # IP address of the remote brick
option remote-subvolume volumenamespace
end-volume
volume server1vol1
type protocol/client
option transport-type tcp/client # for TCP/IP transport
option remote-host 127.0.0.1 # IP address of the remote brick
option remote-subvolume clusterfs1
end-volume
###################
volume bricks
type cluster/unify
option namespace server1
option readdir-force-success on # ignore failed mounts
subvolumes server1vol1
option scheduler rr
option rr.limits.min-free-disk 5 #%
end-volume
========================================
glusterfs-server.vol:
volume clusterfs1
type storage/posix
option directory /volume1
end-volume
#######
volume volumenamespace
type storage/posix
option directory /volume.namespace
end-volume
###
volume clusterfs
type protocol/server
option transport-type tcp/server
subvolumes clusterfs1 volumenamespace
option auth.ip.clusterfs1.allow *
option auth.ip.volumenamespace.allow *
end-volume
