Hello squid users,
I'm trying to figure out, why usually quite stable squid started to
crash rapidly on newly installed server with CentOS 4.4. Time to crash
is up to several seconds, then moderate requests load is applied.
Relevant log records are following:
2007/04/25 17:42:54| Validated 10907 Entries
2007/04/25 17:42:54| store_swap_size = 107484k
2007/04/25 17:42:54| storeLateRelease: released 0 objects
*** glibc detected *** corrupted double-linked list: 0x0a378818 ***
2007/04/25 17:46:31| Beginning Validation Procedure
2007/04/25 17:46:31| Completed Validation Procedure
2007/04/25 17:46:31| Validated 11022 Entries
2007/04/25 17:46:31| store_swap_size = 108704k
2007/04/25 17:46:31| storeLateRelease: released 0 objects
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208784224 (LWP 3473)]
0x004e71f1 in calloc () from /lib/tls/libc.so.6
(gdb) backtrace
#0 0x004e71f1 in calloc () from /lib/tls/libc.so.6
#1 0x080da383 in xcalloc (n=1, sz=20) at util.c:561
#2 0x08096ea2 in memPoolAlloc (pool=0x8b485f8) at MemPool.c:295
#3 0x0804ed93 in aclMatchAclList (list=0x8b53a88, checklist=0x8e65da0)
at acl.c:1442
#4 0x0804f129 in aclCheck (checklist=0x8e65da0) at acl.c:2154
#5 0x080d2e39 in authenticateDigestHandleReply (data=0x8e71a80,
reply=0x8b67e38 "be7acf8db7345b0fd24e106bef45896d")
at digest/auth_digest.c:923
#6 0x08082e38 in helperHandleRead (fd=8, data=0x8b67de0) at
helper.c:769
#7 0x0806db28 in comm_select (msec=10) at comm_generic.c:264
#8 0x0809556d in main (argc=2, argv=0xbfe0b934) at main.c:837
(gdb)
2007/04/25 17:47:14| 1 Duplicate URLs purged.
2007/04/25 17:47:14| 0 Swapfile clashes avoided.
2007/04/25 17:47:14| Took 0.4 seconds (29390.2 objects/sec).
2007/04/25 17:47:14| Beginning Validation Procedure
2007/04/25 17:47:14| Completed Validation Procedure
2007/04/25 17:47:14| Validated 11022 Entries
2007/04/25 17:47:14| store_swap_size = 108704k
2007/04/25 17:47:15| storeLateRelease: released 0 objects
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208554848 (LWP 3485)]
0x004e71f1 in calloc () from /lib/tls/libc.so.6
(gdb) backtrace
#0 0x004e71f1 in calloc () from /lib/tls/libc.so.6
#1 0x080da383 in xcalloc (n=1, sz=20) at util.c:561
#2 0x08096ea2 in memPoolAlloc (pool=0xa2105f8) at MemPool.c:295
#3 0x0804ed93 in aclMatchAclList (list=0xa21bbb8, checklist=0xa52dbf0)
at acl.c:1442
#4 0x0804f129 in aclCheck (checklist=0xa52dbf0) at acl.c:2154
#5 0x080d2e39 in authenticateDigestHandleReply (data=0xa538090,
reply=0xa22fe38 "10e9e6568dcd9030053a9f347a4814bf")
at digest/auth_digest.c:923
#6 0x08082e38 in helperHandleRead (fd=8, data=0xa22fde0) at
helper.c:769
#7 0x0806db28 in comm_select (msec=10) at comm_generic.c:264
#8 0x0809556d in main (argc=2, argv=0xbff037d4) at main.c:837
Configuration options a following:
configure options: '--bindir=/usr/local/sbin'
'--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid'
'--libexecdir=/usr/local/libexec/squid'
'--localstatedir=/usr/local/squid' '--sysconfdir=/usr/local/etc/squid'
'--enable-auth=basic ntlm digest' '--enable-basic-auth-helpers=NCSA PAM
MSNT SMB YP' '--enable-digest-auth-helpers=password'
'--enable-external-acl-helpers=ip_user unix_group wbinfo_group'
'--enable-ntlm-auth-helpers=SMB' '--enable-delay-pools' '--enable-snmp'
'--enable-htcp' '--enable-forw-via-db' '--enable-cache-digests'
'--enable-wccpv2' '--enable-err-languages=English German Russian-1251
Russian-koi8-r' '--enable-default-err-language=English'
'--prefix=/usr/local'
Squid Cache: Version 2.6.STABLE12-20070424 The same behavior is with
2.6.STABLE12 release.
Operation system: CentOS 4.4 with 2.6.9-42.EL #1 Sat Aug 12 09:17:58.
# ldd /usr/local/sbin/squid
libcrypt.so.1 => /lib/libcrypt.so.1 (0x00609000)
libm.so.6 => /lib/tls/libm.so.6 (0x005b0000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00639000)
libc.so.6 => /lib/tls/libc.so.6 (0x00483000)
/lib/ld-linux.so.2 (0x00465000)
I suspect there is some incompatibility between squid and the libc in
use in CentOS 4.4, e.g.:
[root@cache-pril devel]# rpm -qf /lib/tls/libc.so.6
glibc-2.3.4-2.25
What would be my next steps to debug and fix it?
Kind regards,
Anton Golubev
