Hi all,
Bare with me this is a bit long. I've been running
production squid 2.5Stable10 on a Red Hat ES 3
platform for about two weeks now. Yesterday the
server crashed and I haven't been able to find a good
solution or cause for that matter. I'm hoping someone
in this forum can shed some light as to why Squid
crashed and what the correct resolution is. I have
not been able to find a good reason or solution as to
what happened.
We are running Squid with Smart Filter and Heartbeat
on the Linux platform quoted above.
Yesterday around 2:37 Squid just stopped working. The
application was in memory but was just not proxying.
There was no helpful information in syslog. The only
helpful information I found in Squid was in the
cache.log (the details of that log are at the bottom
of this post). Note however, that the failure
happened at 2:37 yesterday and the last cache.log
entry was 1:24.
Now I did some research based on the contents of
cache.log and that lead me to Squid FAQ 11.41 which
doesn't solve the problem. Note that Squid is running
at this time, however if I do nothing than I suppose
the server will just crash again.
Any thoughts, help or ideas would be appreciated. The
cache.log file follows.
Thanks in advance.
Pamela Rock
**** CACHE.LOG ****
2006/02/26 00:00:00| storeDirWriteCleanLogs:
Starting...
2006/02/26 00:00:01| 65536 entries written so far.
2006/02/26 00:00:01| 131072 entries written so far.
2006/02/26 00:00:02| 196608 entries written so far.
2006/02/26 00:00:02| 262144 entries written so far.
2006/02/26 00:00:02| 327680 entries written so far.
2006/02/26 00:00:02| 393216 entries written so far.
2006/02/26 00:00:02| 458752 entries written so far.
2006/02/26 00:00:03| 524288 entries written so far.
2006/02/26 00:00:03| 589824 entries written so far.
2006/02/26 00:00:03| 655360 entries written so far.
2006/02/26 00:00:03| Finished. Wrote 660283
entries.
2006/02/26 00:00:03| Took 1.6 seconds (422376.8
entries/sec).
2006/02/26 00:00:03| logfileRotate:
/usr/local/squid/var/logs/store.log
2006/02/26 00:00:03| logfileRotate:
/usr/local/squid/var/logs/access.log
2006/02/26 01:00:10| Preparing for shutdown after
1150686 requests
2006/02/26 01:00:10| Waiting 30 seconds for active
connections to finish
2006/02/26 01:00:10| FD 33 Closing HTTP connection
2006/02/26 01:00:34| Shutting down...
2006/02/26 01:00:34| FD 34 Closing ICP connection
2006/02/26 01:00:34| Closing unlinkd pipe on FD 28
2006/02/26 01:00:34| storeDirWriteCleanLogs:
Starting...
2006/02/26 01:00:34| 65536 entries written so far.
2006/02/26 01:00:34| 131072 entries written so far.
2006/02/26 01:00:35| 196608 entries written so far.
2006/02/26 01:00:36| 262144 entries written so far.
2006/02/26 01:00:36| 327680 entries written so far.
2006/02/26 01:00:36| 393216 entries written so far.
2006/02/26 01:00:36| 458752 entries written so far.
2006/02/26 01:00:37| 524288 entries written so far.
2006/02/26 01:00:37| 589824 entries written so far.
2006/02/26 01:00:37| 655360 entries written so far.
2006/02/26 01:00:37| Finished. Wrote 660284
entries.
2006/02/26 01:00:37| Took 3.2 seconds (205714.1
entries/sec).
CPU Usage: 2501.330 seconds = 851.930 user + 1649.400
sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 206952
Memory usage for squid via mallinfo():
total space in arena: 81328 KB
Ordinary blocks: 80271 KB 76 blks
Small blocks: 0 KB 6 blks
Holding blocks: 592 KB 3 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 1056 KB
Total in use: 80863 KB 99%
Total free: 1056 KB 1%
2006/02/26 01:00:37| Squid Cache (Version
2.5.STABLE10): Exiting normally.
2006/02/26 04:58:50| Starting Squid Cache version
2.5.STABLE10 for i686-pc-linux-gnu...
2006/02/26 04:58:50| Process ID 4237
2006/02/26 04:58:50| With 1024 file descriptors
available
2006/02/26 04:58:50| Performing DNS Tests...
2006/02/26 04:58:50| Successful DNS name lookup
tests...
2006/02/26 04:58:50| DNS Socket created at 0.0.0.0,
port 32794, FD 9
2006/02/26 04:58:50| Adding nameserver 10.10.0.36 from
/etc/resolv.conf
2006/02/26 04:58:50| Unlinkd pipe opened on FD 28
2006/02/26 04:58:50| Swap maxSize 10240000 KB,
estimated 787692 objects
2006/02/26 04:58:50| Target number of buckets: 39384
2006/02/26 04:58:50| Using 65536 Store buckets
2006/02/26 04:58:50| Max Mem size: 8192 KB
2006/02/26 04:58:50| Max Swap size: 10240000 KB
2006/02/26 04:58:50| Rebuilding storage in
/var/spool/squid (CLEAN)
2006/02/26 04:58:50| Using Least Load store dir
selection
2006/02/26 04:58:50| Set Current Directory to
/usr/local/squid/var/cache
2006/02/26 04:58:50| Loaded Icons.
2006/02/26 04:58:50| Smartfilter: Initializing
SmartFilter
2006/02/26 04:58:50| SmartFilter: SmartFilter Plugin
Library Version 4.1.0.01
2006/02/26 04:58:50| SmartFilter: Trying to start
plugin thread
2006/02/26 04:58:50| SmartFilter: Created
communication thread
2006/02/26 04:58:50| SmartFilter: SmartFilter init:
SmartFilter initialized.
2006/02/26 04:58:50| Accepting HTTP connections at
0.0.0.0, port 3128, FD 33.
2006/02/26 04:58:50| Accepting ICP messages at
0.0.0.0, port 3130, FD 34.
2006/02/26 04:58:50| WCCP Disabled.
2006/02/26 04:58:50| Ready to serve requests.
2006/02/26 04:58:55| Store rebuilding is 0.6%
complete
2006/02/26 04:59:08| Done reading /var/spool/squid
swaplog (660284 entries)
2006/02/26 04:59:08| Finished rebuilding storage from
disk.
2006/02/26 04:59:08| 660284 Entries scanned
2006/02/26 04:59:08| 0 Invalid entries.
2006/02/26 04:59:08| 0 With invalid flags.
2006/02/26 04:59:08| 660284 Objects loaded.
2006/02/26 04:59:08| 0 Objects expired.
2006/02/26 04:59:08| 0 Objects cancelled.
2006/02/26 04:59:08| 0 Duplicate URLs purged.
2006/02/26 04:59:08| 0 Swapfile clashes
avoided.
2006/02/26 04:59:08| Took 18.0 seconds (36693.5
objects/sec).
2006/02/26 04:59:08| Beginning Validation Procedure
2006/02/26 04:59:09| 262144 Entries Validated so
far.
2006/02/26 04:59:09| 524288 Entries Validated so
far.
2006/02/26 04:59:09| Completed Validation Procedure
2006/02/26 04:59:09| Validated 660284 Entries
2006/02/26 04:59:09| store_swap_size = 7847892k
2006/02/26 04:59:09| storeLateRelease: released 0
objects
2006/02/26 09:07:24| Preparing for shutdown after 2316
requests
2006/02/26 09:07:24| Waiting 30 seconds for active
connections to finish
2006/02/26 09:07:24| FD 33 Closing HTTP connection
2006/02/26 09:07:55| Shutting down...
2006/02/26 09:07:55| FD 34 Closing ICP connection
2006/02/26 09:07:55| Closing unlinkd pipe on FD 28
2006/02/26 09:07:55| storeDirWriteCleanLogs:
Starting...
2006/02/26 09:07:56| 65536 entries written so far.
2006/02/26 09:07:56| 131072 entries written so far.
2006/02/26 09:07:56| 196608 entries written so far.
2006/02/26 09:07:56| 262144 entries written so far.
2006/02/26 09:07:56| 327680 entries written so far.
2006/02/26 09:07:56| 393216 entries written so far.
2006/02/26 09:07:56| 458752 entries written so far.
2006/02/26 09:07:56| 524288 entries written so far.
2006/02/26 09:07:56| 589824 entries written so far.
2006/02/26 09:07:56| 655360 entries written so far.
2006/02/26 09:07:56| Finished. Wrote 660289
entries.
2006/02/26 09:07:56| Took 0.3 seconds (2630696.4
entries/sec).
CPU Usage: 13.150 seconds = 5.380 user + 7.770 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 1096
Memory usage for squid via mallinfo():
total space in arena: 64812 KB
Ordinary blocks: 64787 KB 30 blks
Small blocks: 0 KB 6 blks
Holding blocks: 592 KB 3 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 24 KB
Total in use: 65379 KB 101%
Total free: 24 KB 0%
2006/02/26 09:07:56| Squid Cache (Version
2.5.STABLE10): Exiting normally.
2006/02/26 09:09:20| Starting Squid Cache version
2.5.STABLE10 for i686-pc-linux-gnu...
2006/02/26 09:09:20| Process ID 16582
2006/02/26 09:09:20| With 1024 file descriptors
available
2006/02/26 09:09:20| Performing DNS Tests...
2006/02/26 09:09:20| Successful DNS name lookup
tests...
2006/02/26 09:09:20| DNS Socket created at 0.0.0.0,
port 32796, FD 9
2006/02/26 09:09:20| Adding nameserver 10.10.0.36 from
/etc/resolv.conf
2006/02/26 09:09:20| Unlinkd pipe opened on FD 28
2006/02/26 09:09:20| Swap maxSize 10240000 KB,
estimated 787692 objects
2006/02/26 09:09:20| Target number of buckets: 39384
2006/02/26 09:09:20| Using 65536 Store buckets
2006/02/26 09:09:20| Max Mem size: 8192 KB
2006/02/26 09:09:20| Max Swap size: 10240000 KB
2006/02/26 09:09:20| Rebuilding storage in
/var/spool/squid (CLEAN)
2006/02/26 09:09:20| Using Least Load store dir
selection
2006/02/26 09:09:20| Set Current Directory to
/usr/local/squid/var/cache
2006/02/26 09:09:20| Loaded Icons.
2006/02/26 09:09:20| Smartfilter: Initializing
SmartFilter
2006/02/26 09:09:20| SmartFilter: SmartFilter Plugin
Library Version 4.1.0.01
2006/02/26 09:09:20| SmartFilter: Trying to start
plugin thread
2006/02/26 09:09:20| SmartFilter: Created
communication thread
2006/02/26 09:09:20| SmartFilter: SmartFilter init:
SmartFilter initialized.
2006/02/26 09:09:20| Accepting HTTP connections at
0.0.0.0, port 3128, FD 33.
2006/02/26 09:09:20| Accepting ICP messages at
0.0.0.0, port 3130, FD 34.
2006/02/26 09:09:20| WCCP Disabled.
2006/02/26 09:09:20| Ready to serve requests.
2006/02/26 09:09:22| Store rebuilding is 0.6%
complete
2006/02/26 09:09:37| Store rebuilding is 88.1%
complete
2006/02/26 09:09:39| Done reading /var/spool/squid
swaplog (660289 entries)
2006/02/26 09:09:39| Finished rebuilding storage from
disk.
2006/02/26 09:09:39| 660289 Entries scanned
2006/02/26 09:09:39| 0 Invalid entries.
2006/02/26 09:09:39| 0 With invalid flags.
2006/02/26 09:09:39| 660289 Objects loaded.
2006/02/26 09:09:39| 0 Objects expired.
2006/02/26 09:09:39| 0 Objects cancelled.
2006/02/26 09:09:39| 0 Duplicate URLs purged.
2006/02/26 09:09:39| 0 Swapfile clashes
avoided.
2006/02/26 09:09:39| Took 18.3 seconds (35984.1
objects/sec).
2006/02/26 09:09:39| Beginning Validation Procedure
2006/02/26 09:09:39| 262144 Entries Validated so
far.
2006/02/26 09:09:39| 524288 Entries Validated so
far.
2006/02/26 09:09:39| Completed Validation Procedure
2006/02/26 09:09:39| Validated 660289 Entries
2006/02/26 09:09:39| store_swap_size = 7847952k
2006/02/26 09:09:40| storeLateRelease: released 0
objects
2006/02/26 09:43:52| sslReadServer: FD 37: read
failure: (104) Connection reset by peer
2006/02/26 10:15:53| sslReadServer: FD 36: read
failure: (104) Connection reset by peer
2006/02/26 10:36:54| sslReadServer: FD 36: read
failure: (104) Connection reset by peer
2006/02/26 11:07:15| sslReadServer: FD 36: read
failure: (104) Connection reset by peer
2006/02/26 11:12:25| sslReadServer: FD 40: read
failure: (104) Connection reset by peer
2006/02/26 11:17:20| sslReadServer: FD 37: read
failure: (104) Connection reset by peer
2006/02/26 11:24:40| sslReadServer: FD 39: read
failure: (104) Connection reset by peer
2006/02/26 11:27:25| sslReadServer: FD 39: read
failure: (104) Connection reset by peer
2006/02/26 11:28:15| sslReadServer: FD 36: read
failure: (104) Connection reset by peer
2006/02/26 11:37:21| sslReadServer: FD 36: read
failure: (104) Connection reset by peer
2006/02/26 11:47:21| sslReadServer: FD 39: read
failure: (104) Connection reset by peer
2006/02/26 11:52:05| sslReadServer: FD 38: read
failure: (104) Connection reset by peer
2006/02/26 11:52:06| sslReadServer: FD 43: read
failure: (104) Connection reset by peer
2006/02/26 12:00:26| sslReadServer: FD 38: read
failure: (104) Connection reset by peer
2006/02/26 12:01:21| sslReadServer: FD 36: read
failure: (104) Connection reset by peer
2006/02/26 12:06:11| sslReadServer: FD 36: read
failure: (104) Connection reset by peer
2006/02/26 12:09:36| sslReadServer: FD 36: read
failure: (104) Connection reset by peer
2006/02/26 12:13:22| sslReadServer: FD 46: read
failure: (104) Connection reset by peer
2006/02/26 12:16:47| sslReadServer: FD 36: read
failure: (104) Connection reset by peer
2006/02/26 12:17:46| sslReadServer: FD 39: read
failure: (104) Connection reset by peer
2006/02/26 12:21:47| sslReadServer: FD 36: read
failure: (104) Connection reset by peer
2006/02/26 12:22:41| sslReadServer: FD 43: read
failure: (104) Connection reset by peer
2006/02/26 12:29:21| sslReadServer: FD 36: read
failure: (104) Connection reset by peer
2006/02/26 12:30:22| sslReadServer: FD 39: read
failure: (104) Connection reset by peer
2006/02/26 12:37:02| sslReadServer: FD 35: read
failure: (104) Connection reset by peer
2006/02/26 12:55:57| sslReadServer: FD 37: read
failure: (104) Connection reset by peer
2006/02/26 12:55:58| sslReadServer: FD 39: read
failure: (104) Connection reset by peer
2006/02/26 12:57:50| urlParse: Illegal character in
hostname 'n%20wa'
2006/02/26 12:57:50| urlParse: Illegal character in
hostname 'n%20wa'
2006/02/26 13:10:42| sslReadServer: FD 42: read
failure: (104) Connection reset by peer
2006/02/26 13:12:22| sslReadServer: FD 40: read
failure: (104) Connection reset by peer
2006/02/26 13:15:37| sslReadServer: FD 36: read
failure: (104) Connection reset by peer
2006/02/26 13:15:38| sslReadServer: FD 38: read
failure: (104) Connection reset by peer
2006/02/26 13:23:38| sslReadServer: FD 42: read
failure: (104) Connection reset by peer
2006/02/26 13:24:32| sslReadServer: FD 40: read
failure: (104) Connection reset by peer
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
