hi,
i've kept a testing mail service for days, with no other problem than
some low performance under some particular situations.
But now, i'm starting to get problems with libc6.
I've got 4 postfix servers sharing a glusterfs system, version
mainline-2.5, patch 690.
I've also got 6 node servers sharing the storage space that postfix
servers use.
all nodes are still working, and also postfix1. The other ones can't
access the glusterfs once this libc6 message has been logged in the
glusterfs log file.
these are the logs from all the parts:
********postfix1 (it still works) ******
........
2008-03-07 08:49:39 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
22189179: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com
=> -1 (116)
2008-03-07 08:49:46 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
22190414: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com
=> -1 (116)
2008-03-07 08:49:53 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
22191647: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com
=> -1 (116)
2008-03-07 08:49:54 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
22192847: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com
=> -1 (116)
2008-03-07 08:49:55 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
22194013: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com
=> -1 (116)
2008-03-07 08:50:01 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
22195227: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com
=> -1 (116)
2008-03-07 08:50:03 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
22196474: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com
=> -1 (116)
2008-03-07 08:50:04 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
22197676: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com
=> -1 (116)
2008-03-07 08:50:05 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
22198842: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com
=> -1 (116)
2008-03-07 08:50:06 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
22200100: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com
=> -1 (116)
2008-03-07 08:50:07 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
22201281: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com
=> -1 (116)
2008-03-07 08:50:11 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
22202518: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com
=> -1 (116)
2008-03-07 08:50:12 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
22203686: /dummymassiu/tmp/1204836844.P8534.postfix04gluster01.cdmon.com
=> -1 (116)
************************
******postfix2**********
2008-03-06 19:08:09 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
797528: /dummymassiu/tmp/1204830430.P5800.postfix04gluster01.cdmon.com
=> -1 (2)
2008-03-06 19:08:09 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
797528: /dummymassiu/tmp/1204830430.P5800.postfix04gluster01.cdmon.com
=> -1 (2)
2008-03-06 19:08:09 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
797636: /dummymassiu/tmp/1204830431.P5798.postfix04gluster01.cdmon.com
=> -1 (2)
2008-03-06 19:08:09 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
797636: /dummymassiu/tmp/1204830431.P5798.postfix04gluster01.cdmon.com
=> -1 (2)
2008-03-06 19:08:09 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
797684: /dummymassiu/tmp/1204830451.P5663.postfix01gluster01.cdmon.com
=> -1 (2)
2008-03-06 19:08:09 E [unify.c:261:unify_lookup_cbk] ultim: Revalidate
failed for /dummymassiu/tmp/1204830451.P5663.postfix01gluster01.cdmon.com
2008-03-06 19:08:09 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
797686: /dummymassiu/tmp/1204830451.P5663.postfix01gluster01.cdmon.com
=> -1 (2)
TLA Repo Revision: glusterfs--mainline--2.5--patch-690
Time : 2008-03-06 19:08:09
Signal Number : 11
/lib/libc.so.6[0x2b9e19944110]
/usr/lib/libglusterfs.so.0(inode_unref+0x16)[0x2b9e195f1a46]
/usr/lib/glusterfs/1.3.8/xlator/mount/fuse.so[0x2aaaaaeec4f2]
/usr/lib/glusterfs/1.3.8/xlator/mount/fuse.so[0x2aaaaaeec536]
/usr/lib/glusterfs/1.3.8/xlator/mount/fuse.so[0x2aaaaaef0b0f]
/usr/lib/glusterfs/1.3.8/xlator/mount/fuse.so[0x2aaaaaef0cd2]
/usr/lib/glusterfs/1.3.8/xlator/cluster/unify.so(unify_lookup_cbk+0x2f2)[0x2aaaaade5872]
/usr/lib/glusterfs/1.3.8/xlator/cluster/afr.so(afr_sync_ownership_permission+0x473)[0x2aaaaabc7773]
/usr/lib/glusterfs/1.3.8/xlator/cluster/afr.so(afr_check_ctime_version+0x27c)[0x2aaaaabccdbc]
/usr/lib/glusterfs/1.3.8/xlator/cluster/afr.so(afr_lookup_cbk+0x501)[0x2aaaaabce321]
/usr/lib/glusterfs/1.3.8/xlator/protocol/client.so(client_lookup_cbk+0x1ed)[0x2aaaaaab84cd]
/usr/lib/glusterfs/1.3.8/xlator/protocol/client.so(notify+0x8fc)[0x2aaaaaab18bc]
/usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xbf)[0x2b9e195efdcf]
/usr/lib/libglusterfs.so.0(poll_iteration+0x75)[0x2b9e195ef2f5]
[glusterfs](main+0x64f)[0x40272f]
/lib/libc.so.6(__libc_start_main+0xda)[0x2b9e199314ca]
[glusterfs][0x401c9a]
***********************
********postfix3**************
2008-03-06 18:33:23 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
399453: /dummymassiu/tmp/1204828340.P2724.postfix01gluster01.cdmon.com
=> -1 (2)
2008-03-06 18:33:23 W [fuse-bridge.c:378:fuse_entry_cbk] glusterfs-fuse:
401349: /dummymassiu/tmp/1204828380.P2788.postfix04gluster01.cdmon.com
=> 63434 Rehashing f000/0
2008-03-06 18:33:23 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
401353: /dummymassiu/tmp/1204828380.P2788.postfix04gluster01.cdmon.com
=> -1 (2)
2008-03-06 18:33:23 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
401353: /dummymassiu/tmp/1204828380.P2788.postfix04gluster01.cdmon.com
=> -1 (2)
TLA Repo Revision: glusterfs--mainline--2.5--patch-690
Time : 2008-03-06 18:33:23
Signal Number : 11
/lib/libc.so.6[0x2b9e512a3110]
/lib/libc.so.6[0x2b9e512de7ff]
/lib/libc.so.6[0x2b9e512e0789]
/lib/libc.so.6(calloc+0x96)[0x2b9e512e1e46]
/usr/lib/glusterfs/1.3.8/xlator/cluster/unify.so(unify_lookup+0x54)[0x2aaaaade2c14]
/usr/lib/glusterfs/1.3.8/xlator/mount/fuse.so[0x2aaaaaeef3f7]
/usr/lib/glusterfs/1.3.8/xlator/mount/fuse.so[0x2aaaaaeeface]
/lib/libpthread.so.0[0x2b9e51164f1a]
/lib/libc.so.6(__clone+0x72)[0x2b9e5133d5d2]
****************************
******postfix4**********
2008-03-06 20:54:15 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
2606669: /dummymassiu/tmp/1204836803.P11411.postfix01gluster01.cdmon.com
=> -1 (2)
2008-03-06 20:54:15 E [fuse-bridge.c:447:fuse_entry_cbk] glusterfs-fuse:
2606669: /dummymassiu/tmp/1204836803.P11411.postfix01gluster01.cdmon.com
=> -1 (2)
2008-03-06 20:54:15 W [fuse-bridge.c:378:fuse_entry_cbk] glusterfs-fuse:
2606806: /dummymassiu/tmp/1204836832.P10986.postfix01gluster01.cdmon.com
=> 103147 Rehashing 9000/0
2008-03-06 20:54:15 W [fuse-bridge.c:378:fuse_entry_cbk] glusterfs-fuse:
2606807: /dummymassiu/tmp/1204836832.P10986.postfix01gluster01.cdmon.com
=> 103147 Rehashing 9000/0
TLA Repo Revision: glusterfs--mainline--2.5--patch-690
Time : 2008-03-06 20:54:15
Signal Number : 11
/lib/libc.so.6[0x2b6377f4e110]
/usr/lib/glusterfs/1.3.8/xlator/cluster/unify.so(unify_lookup_cbk+0x2c)[0x2aaaaade55ac]
/usr/lib/glusterfs/1.3.8/xlator/cluster/afr.so(afr_lookup_cbk+0x2cc)[0x2aaaaabce0ec]
/usr/lib/glusterfs/1.3.8/xlator/protocol/client.so(client_lookup_cbk+0x1ed)[0x2aaaaaab84cd]
/usr/lib/glusterfs/1.3.8/xlator/protocol/client.so(notify+0x8fc)[0x2aaaaaab18bc]
/usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xbf)[0x2b6377bf9dcf]
/usr/lib/libglusterfs.so.0(poll_iteration+0x75)[0x2b6377bf92f5]
[glusterfs](main+0x64f)[0x40272f]
/lib/libc.so.6(__libc_start_main+0xda)[0x2b6377f3b4ca]
[glusterfs][0x401c9a]
**********************
*******node1*********
2008-03-06 16:54:15 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 17:01:38 E [protocol.c:259:gf_block_unserialize_transport]
ultim: EOF from peer (192.168.1.219:1021)
2008-03-06 17:01:38 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 17:01:38 E [protocol.c:259:gf_block_unserialize_transport]
ultim: EOF from peer (192.168.1.219:1023)
2008-03-06 17:01:38 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 18:33:22 E [protocol.c:259:gf_block_unserialize_transport]
ultim: EOF from peer (192.168.1.34:1019)
2008-03-06 18:33:22 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 18:33:22 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 18:33:24 E [server-protocol.c:183:generic_reply] ultim:
transport_writev failed
2008-03-06 18:33:24 E [server-protocol.c:183:generic_reply] ultim:
transport_writev failed
2008-03-06 18:33:24 E [server-protocol.c:183:generic_reply] ultim:
transport_writev failed
2008-03-06 19:08:11 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 19:08:11 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 20:54:17 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 20:54:17 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
***********************
***********node2***********
2008-03-06 17:01:37 E [protocol.c:259:gf_block_unserialize_transport]
ultim: EOF from peer (192.168.1.219:1022)
2008-03-06 17:01:37 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 17:32:45 E [server-protocol.c:4862:server_lk] espai: Unknown
lock type: 20688!
2008-03-06 17:32:45 E [server-protocol.c:4862:server_lk] espai: Unknown
lock type: 20688!
2008-03-06 17:33:21 E [server-protocol.c:4862:server_lk] espai: Unknown
lock type: 2080!
2008-03-06 17:33:21 E [server-protocol.c:4862:server_lk] espai: Unknown
lock type: 2080!
2008-03-06 18:04:41 E [server-protocol.c:4862:server_lk] espai: Unknown
lock type: -27456!
2008-03-06 18:04:41 E [server-protocol.c:4862:server_lk] espai: Unknown
lock type: -27456!
2008-03-06 18:33:23 E [protocol.c:259:gf_block_unserialize_transport]
ultim: EOF from peer (192.168.1.34:1018)
2008-03-06 18:33:23 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 18:33:23 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 19:08:11 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 19:08:11 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 20:54:17 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
2008-03-06 20:54:17 C [tcp.c:87:tcp_disconnect] ultim: connection
disconnected
***************************
