More information on cache.log 2012/08/20 14:46:35| WARNING: swapfile header inconsistent with available data FATAL: Received Segment Violation...dying. 2012/08/20 14:46:35| storeDirWriteCleanLogs: Starting... 2012/08/20 14:46:35| WARNING: Closing open FD 36 2012/08/20 14:46:35| 65536 entries written so far. 2012/08/20 14:46:35| 131072 entries written so far. 2012/08/20 14:46:35| 196608 entries written so far. -----Original Message----- From: Julie Xu [mailto:J.Xu@xxxxxxxxxx] Sent: Tuesday, 21 August 2012 7:49 AM To: Eliezer Croitoru; squid-users@xxxxxxxxxxxxxxx Subject: RE: squid on redhat6 crash Eliezer Great thanks for the reply. From briefly dump information I only get is: Crash comman: is squid -f /etc/squid/squid.conf Reason: is Process /usr/sbin/squid was killed by signal 6 (SIGABRT) Kernel: is 2.6.32-279.2.1.el6.x86_64 (Red Hat Enterprise Linux Server release 6.3 (Santiago)) Pwd: /var/cache/squid Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10485760 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 256718 256718 processes Max open files 1024 4096 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 256718 256718 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Below to answer your question: >you mean load by req\s?memory?cpu?disk i\o? The load I mean is when squid client increase; I found it only crash between 9am-9pm, did not crash at night time. >how many requests per sec? Now, I can not check, it is out of service now. >squid 3.1 self compiled or from rpm? It is from system rpm. >using any special helpers?how many child process(for helpers)? Squid_ldap_auth, 10 child process, do not anything else. >"squid -v" output? ]# squid -v Squid Cache: Version 3.1.10 configure options: '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '--target=x86_64-redhat-linux-gnu' '--program-prefix=' '--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/bin' '--sbindir=/usr/sbin' '--sysconfdir=/etc' '--datadir=/usr/share' '--includedir=/usr/include' '--libdir=/usr/lib64' '--libexecdir=/usr/libexec' '--sharedstatedir=/var/lib' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--exec_prefix=/usr' '--libexecdir=/usr/lib64/squid' '--localstatedir=/var' '--datadir=/usr/share/squid' '--sysconfdir=/etc/squid' '--with-logdir=$(localstatedir)/log/squid' '--with-pidfile=$(localstatedir)/run/squid.pid' '--disable-dependency-tracking' '--enable-arp-acl' '--enable-follow-x-forwarded-for' '--enable-auth=basic,digest,ntlm,negotiate' '--enable-basic-auth-helpers=LDAP,MSNT,NCSA,PAM,SMB,YP,getpwnam,multi-domain-NTLM,SASL,DB,POP3,squid_radius_auth' '--enable-ntlm-auth-helpers=smb_lm,no_check,fakeauth' '--enable-digest-auth-helpers=password,ldap,eDirectory' '--enable-negotiate-auth-helpers=squid_kerb_auth' '--enable-external-acl-helpers=ip_user,ldap_group,session,unix_group,wbinfo_group' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-epoll' '--enable-icap-client' '--enable-ident-lookups' '--enable-linux-netfilter' '--enable-referer-log' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,ufs' '--enable-useragent-log' '--enable-wccpv2' '--enable-esi' '--with-aio' '--with-default-user=squid' '--with-filedescriptors=16384' '--with-dl' '--with-openssl' '--with-pthreads' 'build_alias=x86_64-redhat-linux-gnu' 'host_alias=x86_64-redhat-linux-gnu' 'target_alias=x86_64-redhat-linux-gnu' 'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fpie' 'LDFLAGS=-pie' 'CXXFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fpie' --with-squid=/builddir/build/BUILD/squid-3.1.10 >squid.conf ? Related part: auth_param basic program /usr/lib64/squid/squid_ldap_auth ........ auth_param basic children 10 auth_param basic realm xx auth_param basic credentialsttl 15 minutes authenticate_ip_ttl 5 minutes acl all src all icp_access deny all acl_uses_indirect_client off log_uses_indirect_client off http_port xxx hierarchy_stoplist cgi-bin ? squid-internal- cache_mem 2048 MB maximum_object_size_in_memory 1 MB memory_replacement_policy heap GDSF cache_replacement_policy heap LFUDA cache_dir aufs xxxx maximum_object_size 64 MB cache_store_log none dns_retransmit_interval 1 second dns_timeout 1 minute ipcache_size 8192 ipcache_low 95 ipcache_high 98 memory_pools_limit none forwarded_for off uri_whitespace deny coredump_dir /var/cache/squid pipeline_prefetch on sleep_after_fork 500000 >transparent proxy? no >"ulimit -Sa" output? ore file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 256718 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited >"ulimit -Ha" output? # ulimit -Ha core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 256718 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 4096 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) unlimited cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited many regards Julie -----Original Message----- From: Eliezer Croitoru [mailto:eliezer@xxxxxxxxxxxx] Sent: Tuesday, 21 August 2012 12:33 AM To: squid-users@xxxxxxxxxxxxxxx Subject: Re: squid on redhat6 crash On 8/20/2012 4:01 PM, Julie Xu wrote: > Hi > > Used redhat6' squid 3.1, and the system is crashing when there is load increased. > you mean load by req\s?memory?cpu?disk i\o? how many requests per sec? > Aug 20 14:30:01 proxy-server squid[2450]: Squid Parent: child process > 6591 exited due to signal 6 with status 0 Aug 20 14:30:04 proxy-server > squid[2450]: Squid Parent: child process 7091 started Aug 20 14:46:40 > proxy-server abrt[22338]: Saved core dump of pid 7091 > (/usr/sbin/squid) to /var/spool/abrt/ccpp-2012-08-20-14:46:36-7091 > (668131328 bytes) > > Briefly look the dump directory, only thing I found is the direct is the cache direct, there is a coredump file, try to use crash to read it, not successful. > > Could anyone advise me what is possible reason for the crash, and the solution? squid 3.1 self compiled or from rpm? using any special helpers?how many child process(for helpers)? "squid -v" output? squid.conf ? transparent proxy? "ulimit -Sa" output? "ulimit -Ha" output? the more data you will provide us we can pinpoint the reason for the problem. Regards, Eliezer > > Thanks in advance > > Julie > -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer <at> ngtech.co.il
