Hi, I am new to the mailing list and have the following issue:
Our squid crash every night between 01:00 and 02:00 CET.
Mar 15 01:50:33 srv-app-904 (squid-1): Bungled (null) line 8: icap_retry deny all
Mar 15 01:50:35 srv-app-904 squid[3589]: Squid Parent: (squid-1) process 3592 exited with status 1
Mar 15 01:50:38 srv-app-904 squid[3589]: Squid Parent: (squid-1) process 4119 started
Mar 15 01:51:32 srv-app-904 (squid-1): Bungled (null) line 8: icap_retry deny all
Mar 15 01:51:32 srv-app-904 squid[3589]: Squid Parent: (squid-1) process 4119 exited with status 1
Mar 15 01:51:35 srv-app-904 squid[3589]: Squid Parent: (squid-1) process 4141 started
Mar 15 02:13:43 srv-app-904 (squid-1): Bungled (null) line 8: icap_retry deny all
Mar 15 02:13:43 srv-app-904 squid[3589]: Squid Parent: (squid-1) process 4141 exited with status 1
Mar 15 02:13:46 srv-app-904 squid[3589]: Squid Parent: (squid-1) process 4193 started
Cronlog told me that:
Mar 15 01:01:01 srv-app-904 CROND[4079]: (root) CMD (run-parts /etc/cron.hourly)
Mar 15 01:01:01 srv-app-904 run-parts(/etc/cron.hourly)[4079]: starting 0anacron
Mar 15 01:01:01 srv-app-904 anacron[4090]: Anacron started on 2013-03-15
Mar 15 01:01:01 srv-app-904 anacron[4090]: Jobs will be executed sequentially
Mar 15 01:01:01 srv-app-904 anacron[4090]: Normal exit (0 jobs run)
Mar 15 01:01:01 srv-app-904 run-parts(/etc/cron.hourly)[4092]: finished 0anacron
Mar 15 01:01:01 srv-app-904 run-parts(/etc/cron.hourly)[4079]: starting mcelog.cron
Mar 15 01:01:01 srv-app-904 run-parts(/etc/cron.hourly)[4100]: finished mcelog.cron
Mar 15 01:10:01 srv-app-904 CROND[4105]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 15 01:20:01 srv-app-904 CROND[4108]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 15 01:30:01 srv-app-904 CROND[4111]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 15 01:40:01 srv-app-904 CROND[4114]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 15 01:50:01 srv-app-904 CROND[4117]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 15 02:00:01 srv-app-904 CROND[4164]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 15 02:01:01 srv-app-904 CROND[4167]: (root) CMD (run-parts /etc/cron.hourly)
Mar 15 02:01:01 srv-app-904 run-parts(/etc/cron.hourly)[4167]: starting 0anacron
Mar 15 02:01:01 srv-app-904 anacron[4178]: Anacron started on 2013-03-15
Mar 15 02:01:01 srv-app-904 anacron[4178]: Jobs will be executed sequentially
Mar 15 02:01:01 srv-app-904 anacron[4178]: Normal exit (0 jobs run)
Mar 15 02:01:01 srv-app-904 run-parts(/etc/cron.hourly)[4180]: finished 0anacron
Mar 15 02:01:01 srv-app-904 run-parts(/etc/cron.hourly)[4167]: starting mcelog.cron
Mar 15 02:01:01 srv-app-904 run-parts(/etc/cron.hourly)[4188]: finished mcelog.cron
Mar 15 02:10:01 srv-app-904 CROND[4191]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 15 02:20:01 srv-app-904 CROND[4216]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 15 02:30:01 srv-app-904 CROND[4219]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 15 02:40:01 srv-app-904 CROND[4222]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 15 02:50:01 srv-app-904 CROND[4225]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Mar 15 03:00:01 srv-app-904 CROND[4228]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Logrotation this:
Mar 15 03:10:01 srv-app-904 run-parts(/etc/cron.daily)[4253]: starting logrotate
Mar 15 03:10:17 srv-app-904 run-parts(/etc/cron.daily)[4280]: finished logrotate
And here the interesting part from cache.log (with minus in the memory usage????):
2013/03/15 01:50:33 kid1| Closing HTTP port 160.85.104.14:8080
2013/03/15 01:50:33 kid1| storeDirWriteCleanLogs: Starting...
2013/03/15 01:50:33 kid1| 65536 entries written so far.
2013/03/15 01:50:33 kid1| 131072 entries written so far.
2013/03/15 01:50:33 kid1| 196608 entries written so far.
2013/03/15 01:50:33 kid1| 262144 entries written so far.
2013/03/15 01:50:33 kid1| 327680 entries written so far.
2013/03/15 01:50:33 kid1| 393216 entries written so far.
2013/03/15 01:50:33 kid1| 458752 entries written so far.
2013/03/15 01:50:33 kid1| 524288 entries written so far.
2013/03/15 01:50:33 kid1| 589824 entries written so far.
2013/03/15 01:50:33 kid1| 655360 entries written so far.
2013/03/15 01:50:33 kid1| 720896 entries written so far.
2013/03/15 01:50:33 kid1| 786432 entries written so far.
2013/03/15 01:50:33 kid1| Finished. Wrote 799896 entries.
2013/03/15 01:50:33 kid1| Took 0.16 seconds (5155895.89 entries/sec).
FATAL: Bungled (null) line 8: icap_retry deny all
Squid Cache (Version 3.2.8): Terminated abnormally.
CPU Usage: 1200.378 seconds = 742.532 user + 457.845 sys
Maximum Resident Size: 10759744 KB
Page faults with physical i/o: 12
Memory usage for squid via mallinfo():
total space in arena: -1523816 KB
Ordinary blocks: -1587068 KB 98685 blks
Small blocks: 0 KB 1 blks
Holding blocks: 40796 KB 10 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 63251 KB
Total in use: -1546272 KB 101%
Total free: 63251 KB -3%
2013/03/15 01:50:38 kid1| Starting Squid Cache version 3.2.8 for x86_64-unknown-linux-gnu...
2013/03/15 01:50:38 kid1| Process ID 4119
2013/03/15 01:50:38 kid1| Process Roles: worker
2013/03/15 01:50:38 kid1| With 65535 file descriptors available
2013/03/15 01:50:38 kid1| Initializing IP Cache...
2013/03/15 01:50:38 kid1| DNS Socket created at 0.0.0.0, FD 7
2013/03/15 01:50:38 kid1| Adding nameserver 160.85.192.100 from squid.conf
2013/03/15 01:50:38 kid1| Logfile: opening log daemon:/var/log/squid/access.log
2013/03/15 01:50:38 kid1| Logfile Daemon: opening log /var/log/squid/access.log
2013/03/15 01:50:38 kid1| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2013/03/15 01:50:38 kid1| Store logging disabled
2013/03/15 01:50:38 kid1| Swap maxSize 25600000 + 2097152 KB, estimated 2130550 objects
2013/03/15 01:50:38 kid1| Target number of buckets: 106527
The Cache settings are as followed:
cache_effective_user squid
cache_effective_group squid
cache_mem 2048 MB
cache_dir aufs /var/cache/squid 25000 64 256
maximum_object_size_in_memory 512 KB
System Settings:
- CentOS release 6.4 (Final)
- uname -a: Linux srv-app-904 2.6.32-358.2.1.el6.x86_64 #1 SMP Wed Mar 13 00:26:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
- rpm package: squid-3.2.8-1.el6.x86_64
- MemTotal: 8061980 kB
Thanks!
Remo
