Greetings,
A Squid server (RHEL-5.2 64-bit) is serving around 1000 users with
NTLM (winbindd),
Basic, wbinfo, and URL rewriter (Websense). Squid has randomly crashed with
the following messages:
FATAL: Received Segment Violation...dying.
2010/01/20 15:53:29| storeDirWriteCleanLogs: Starting...
2010/01/20 15:53:29| WARNING: Closing open FD 183
2010/01/20 15:53:29| commSetEvents: epoll_ctl(EPOLL_CTL_DEL): failed
on fd=183: (1) Operation not permitted
2010/01/20 15:53:29| 65536 entries written so far.
2010/01/20 15:53:29| 131072 entries written so far.
...
...
2010/01/20 15:53:35| Finished. Wrote 10511684 entries.
2010/01/20 15:53:35| Took 6.2 seconds (1702503.8 entries/sec).
CPU Usage: 3406.086 seconds = 2431.073 user + 975.013 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 3
Memory usage for squid via mallinfo():
total space in arena: -1092812 KB
Ordinary blocks: -1096748 KB 3986 blks
Small blocks: 0 KB 24 blks
Holding blocks: 24108 KB 8 blks
Free Small blocks: 1 KB
Free Ordinary blocks: 3935 KB
Total in use: -1072640 KB 100%
Total free: 3936 KB 0%
2010/01/20 15:54:48| Starting Squid Cache version 2.7.STABLE7 for
x86_64-unknown-linux-gnu...
Ran gdb over the core dump file and got this result:
# gdb /usr/local/squid/sbin/squid core.30068
GNU gdb Red Hat Linux (6.5-37.el5rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db
library "/lib64/libthread_db.so.1".
Reading symbols from /lib64/libcrypt.so.1...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpthread.so.0...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libm.so.6...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libnsl.so.1...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libc.so.6...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libnss_files.so.2...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /lib64/libresolv.so.2...done.
Loaded symbols for /lib64/libresolv.so.2
Core was generated by `(squid)'.
Program terminated with signal 6, Aborted.
#0 0x000000376b630215 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x000000376b630215 in raise () from /lib64/libc.so.6
#1 0x000000376b631cc0 in abort () from /lib64/libc.so.6
#2 0x0000000000474515 in death (sig=<value optimized out>) at tools.c:327
#3 <signal handler called>
#4 0x000000000044271a in httpHeaderGetEntry (hdr=0x141868f8,
pos=0x7fff2c609774) at HttpHeader.c:595
#5 0x00000000004427fe in httpHeaderFindEntry (hdr=0x141868f8,
id=HDR_CONTENT_TYPE) at HttpHeader.c:619
#6 0x0000000000443037 in httpHeaderGetStr (hdr=0x141868f8,
id=HDR_CONTENT_TYPE) at HttpHeader.c:1086
#7 0x00000000004075b9 in aclMatchAclList (list=0x2aaac246fa40,
checklist=0x2aaad3524738) at acl.c:2010
#8 0x0000000000407d82 in aclCheck (checklist=0x2aaad3524738) at acl.c:2178
#9 0x000000000042daae in externalAclHandleReply (data=0x2aaab26015d0,
reply=0x2aaac6951e38 "") at external_acl.c:985
#10 0x000000000043cd65 in helperHandleRead (fd=<value optimized out>,
data=<value optimized out>) at helper.c:769
#11 0x0000000000427511 in comm_select (msec=<value optimized out>)
at comm_generic.c:264
#12 0x000000000044f7a6 in main (argc=<value optimized out>,
argv=0x7fff2c609a58) at main.c:863
The configure options read
./configure --prefix=/usr/local/squid --enable-async-io --enable-snmp --enab
le-epoll --enable-storeio=aufs null --enable-removal-policies=heap lru
--enable-
large-cache-files --with-large-files --disable-poll --disable-select
--disable-c
arp --disable-wccp --disable-ident-lookups --with-maxfd=32768
--enable-ntlm-auth
-helpers=SMB --enable-external-acl-helpers=wbinfo_group ldap_group
--enable-auth
=basic ntlm --enable-basic-auth-helpers=LDAP --enable-err-languages=English
Snippet of squid.conf
emulate_httpd_log on
buffered_logs on
debug_options ALL,1
logfile_daemon /usr/local/squid/libexec/logfile-daemon
access_log daemon:/var/log/squid/access.log
shutdown_lifetime 10 seconds
cache_dir aufs /var/spool/squid/cache01 62500 32 256
cache_dir aufs /var/spool/squid/cache02 62500 32 256
cache_dir aufs /var/spool/squid/cache03 62500 32 256
cache_dir aufs /var/spool/squid/cache04 62500 32 256
cache_replacement_policy heap GDSF
memory_replacement_policy heap GDSF
cache_log /var/log/squid/cache.log
cache_store_log none
pid_filename /var/run/squid.pid
coredump_dir /var/spool/squid/crash
log_icp_queries off
client_db on
half_closed_clients off
cache_mem 512 MB
maximum_object_size 768000 KB
maximum_object_size_in_memory 96 KB
memory_pools off
forwarded_for off
snmp_port 1601
snmp_incoming_address 0.0.0.0
snmp_outgoing_address 255.255.255.255
auth_param ntlm program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
auth_param ntlm children 30
auth_param ntlm keep_alive on
auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic
auth_param basic children 5
auth_param basic realm User Authentication
external_acl_type ads-group children=20 %LOGIN
/usr/local/squid/libexec/wbinfo_group.pl
url_rewrite_children 50
redirector_bypass off
url_rewrite_program /opt/Websense/bin/WsRedtor
Does anyone have any idea how to fix this problem?
Many Thanks
myOcella
