Re: 3.1.3 crash on volume start

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Sadly no go with 3.1.4qa3:

patchset: v3.1.4qa3
signal received: 11
time of crash: 2011-04-12 16:16:34
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.4qa3
/lib64/libc.so.6(+0x33140)[0x7f68d48c8140]
/opt/glusterfs/3.1.4qa3/lib64/glusterfs/3.1.4qa3/xlator/mgmt/glusterd.so(+0x3323b)[0x7f68d311e23b]
/opt/glusterfs/3.1.4qa3/lib64/glusterfs/3.1.4qa3/xlator/mgmt/glusterd.so(glusterd_op_sm+0xee)[0x7f68d311d6ce]
/opt/glusterfs/3.1.4qa3/lib64/glusterfs/3.1.4qa3/xlator/mgmt/glusterd.so(glusterd_handle_cli_start_volume+0x8f)[0x7f68d310c91f]
/opt/glusterfs/3.1.4qa3/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x28f)[0x7f68d526e01f]
/opt/glusterfs/3.1.4qa3/lib64/libgfrpc.so.0(rpcsvc_notify+0x123)[0x7f68d526e653]
/opt/glusterfs/3.1.4qa3/lib64/libgfrpc.so.0(rpc_transport_notify+0x2d)[0x7f68d527221d]
/opt/glusterfs/3.1.4qa3/lib64/glusterfs/3.1.4qa3/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7f68d2ee3b44]
/opt/glusterfs/3.1.4qa3/lib64/glusterfs/3.1.4qa3/rpc-transport/socket.so(socket_event_handler+0xb3)[0x7f68d2ee3de3]
/opt/glusterfs/3.1.4qa3/lib64/libglusterfs.so.0(+0x37d42)[0x7f68d54b2d42]
/opt/glusterfs/3.1.4qa3/sbin/glusterd(main+0x263)[0x4054b3]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f68d48b3e5d]
/opt/glusterfs/3.1.4qa3/sbin/glusterd[0x402db9]

On 04/12/2011 04:07 PM, Mark "Naoki" Rogers wrote:
I had an interesting situation with a (3x3) distribute+replicate setup.
One of the machines died (memory problem, hard lock) and gluster didn't
exactly like it (although that's another story).

Anyway I decided to keep on truckin' and use a 5 node distribute (since
this is mainly a network test anyway) and the volume creation was ok but
startup failed with the old "cli op: 16" message (note: would be nice if
that actually said lock file issue if that's what it is - seems to be
solved by a restart anyway).

I restarted glusterd but everytime I try and "volume start" it just
crashes immediately:

[2011-04-12 15:36:39.861099] I
[glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid:
af4007ac-804c-433f-a6e9-0a7d6cafe432, host: pdbsearch12
[2011-04-12 15:36:39.861123] I
[glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid:
5e4ef932-cca7-4440-a8e3-4415b84c2307, host: pdbsearch13
[2011-04-12 15:36:39.861148] I
[glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid:
df6c1fb1-cf5d-4a6b-ba15-fb26319e37db, host: pdbsearch14
[2011-04-12 15:36:39.861172] I
[glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid:
2cc51957-2c92-4d24-a53c-c7ded19e80b7, host: pdbsearch15
[2011-04-12 15:36:39.861695] I
[glusterd-rpc-ops.c:626:glusterd3_1_friend_update_cbk] 0-glusterd:
Received ACC from uuid:
[2011-04-12 15:36:39.861737] I
[glusterd-rpc-ops.c:626:glusterd3_1_friend_update_cbk] 0-glusterd:
Received ACC from uuid:
[2011-04-12 15:36:39.861776] I
[glusterd-rpc-ops.c:626:glusterd3_1_friend_update_cbk] 0-glusterd:
Received ACC from uuid:
[2011-04-12 15:36:39.861809] I
[glusterd-rpc-ops.c:626:glusterd3_1_friend_update_cbk] 0-glusterd:
Received ACC from uuid:
[2011-04-12 15:36:45.725525] I
[glusterd-handler.c:1088:glusterd_handle_cli_start_volume] 0-glusterd:
Received start vol reqfor volume distribute
[2011-04-12 15:36:45.725677] I [glusterd-utils.c:242:glusterd_lock]
0-glusterd: Cluster lock held by 1b6130e7-f83e-4ec6-90f7-46706dbffc4f
[2011-04-12 15:36:45.725702] I
[glusterd-handler.c:825:glusterd_op_txn_begin] 0-glusterd: Acquired
local lock
pending frames:

patchset: v3.1.3
signal received: 11
time of crash: 2011-04-12 15:36:45
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.3
/lib64/libc.so.6(+0x33140)[0x7fc7e2337140]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(+0x2a3e9)[0x7fc7e0b853e9]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(glusterd_op_sm+0xee)[0x7fc7e0b8b8ce]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(glusterd_handle_cli_start_volume+0x8a)[0x7fc7e0b8197a]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x271)[0x7fc7e2cde061]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpcsvc_notify+0x93)[0x7fc7e2cde183]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_transport_notify+0x2d)[0x7fc7e2cde77d]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7fc7e09545b4]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_handler+0x19d)[0x7fc7e095476d]
/opt/glusterfs/3.1.3/lib64/libglusterfs.so.0(+0x39422)[0x7fc7e2f23422]
/opt/glusterfs/3.1.3/sbin/glusterd(main+0x245)[0x405235]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fc7e2322e5d]
/opt/glusterfs/3.1.3/sbin/glusterd[0x402ea9]

Restarted all nodes and tried again:

patchset: v3.1.3
signal received: 11
time of crash: 2011-04-12 15:47:44
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.3
/lib64/libc.so.6(+0x33140)[0x7fb8f65bb140]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(+0x2a3e9)[0x7fb8f4e093e9]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(glusterd_op_sm+0xee)[0x7fb8f4e0f8ce]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/xlator/mgmt/glusterd.so(glusterd_handle_cli_start_volume+0x8a)[0x7fb8f4e0597a]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x271)[0x7fb8f6f62061]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpcsvc_notify+0x93)[0x7fb8f6f62183]
/opt/glusterfs/3.1.3/lib64/libgfrpc.so.0(rpc_transport_notify+0x2d)[0x7fb8f6f6277d]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7fb8f4bd85b4]
/opt/glusterfs/3.1.3/lib64/glusterfs/3.1.3/rpc-transport/socket.so(socket_event_handler+0x19d)[0x7fb8f4bd876d]
/opt/glusterfs/3.1.3/lib64/libglusterfs.so.0(+0x39422)[0x7fb8f71a7422]
/opt/glusterfs/3.1.3/sbin/glusterd(main+0x245)[0x405235]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7fb8f65a6e5d]
/opt/glusterfs/3.1.3/sbin/glusterd[0x402ea9]


I'm going to probably give a 3.1 (qa4) and/or 3.2 (qa11) release a try
just in the meantime.
Cheers.

_______________________________________________
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxx
http://lists.nongnu.org/mailman/listinfo/gluster-devel




[Index of Archives]     [Gluster Users]     [Ceph Users]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux