Here is the output from df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 7.9G 1.5G 6.1G 20% /
/dev/sda1 99M 12M 83M 13% /boot
/dev/sda6 109G 45G 59G 44% /cache
none 1.8G 0 1.8G 0% /dev/shm
/dev/sda2 16G 13G 2.6G 83% /var
It looks like squid started crashing only 2 days ago after the logrotate,
here is the script:
/var/log/squid/access.log {
daily
rotate 30
copytruncate
compress
notifempty
missingok
}
/var/log/squid/cache.log {
daily
rotate 30
copytruncate
compress
notifempty
missingok
}
/var/log/squid/store.log {
daily
rotate 30
copytruncate
compress
notifempty
missingok
# This script asks squid to rotate its logs on its own.
# Restarting squid is a long process and it is not worth
# doing it just to rotate logs
# postrotate
# /usr/sbin/squid -k rotate
# endscript
}
It has been working fine for a long time, so I am not quite sure why it is
now crashing. Could it be that our internet traffic has increased and the
logs are getting so large than when they are rotating and zipping there
isn't enough space on /var?
From: trainier@xxxxxxxxxx
To: squid-users@xxxxxxxxxxxxxxx
Subject: Re: Crashed squid 2.5.STABLE11
Date: Tue, 11 Oct 2005 16:21:47 -0400
Sorry. That's `df -h` as opposed to `du -h`.
Tim Rainier
Information Services, Kalsec, INC
trainier@xxxxxxxxxx
trainier@xxxxxxxxxx
10/11/2005 03:38 PM
To
squid-users@xxxxxxxxxxxxxxx
cc
Subject
Re: Crashed squid 2.5.STABLE11
First, and foremost, I would hesitate rotating the store log. Henrik and
probably several others, can verify that notion.
Second, do a `du -h` and email the output back.
Tim Rainier
Information Services, Kalsec, INC
trainier@xxxxxxxxxx
"Lucia Di Occhi" <saint_lucy@xxxxxxxxxxx>
10/11/2005 02:29 PM
To
trainier@xxxxxxxxxx, squid-users@xxxxxxxxxxxxxxx
cc
Subject
Re: Crashed squid 2.5.STABLE11
Sure it does and I keep 30 days worth of logs:
squid[~]ls -alh /var/log/squid/
total 12G
drwxr-x--- 2 squid squid 4.0K Oct 11 04:07 .
drwxr-xr-x 12 root root 4.0K Oct 9 04:04 ..
-rw-r--r-- 1 squid squid 365M Oct 11 14:28 access.log
-rw-r--r-- 1 squid squid 47M Oct 2 04:02 access.log.10.gz
-rw-r--r-- 1 squid squid 140M Oct 1 04:04 access.log.11.gz
-rw-r--r-- 1 squid squid 172M Sep 30 04:04 access.log.12.gz
-rw-r--r-- 1 squid squid 174M Sep 29 04:04 access.log.13.gz
-rw-r--r-- 1 squid squid 155M Sep 28 04:04 access.log.14.gz
-rw-r--r-- 1 squid squid 174M Sep 27 04:04 access.log.15.gz
-rw-r--r-- 1 squid squid 90M Sep 26 04:03 access.log.16.gz
-rw-r--r-- 1 squid squid 66M Sep 25 04:03 access.log.17.gz
-rw-r--r-- 1 squid squid 146M Sep 24 04:04 access.log.18.gz
-rw-r--r-- 1 squid squid 158M Sep 23 04:04 access.log.19.gz
-rw-r--r-- 1 squid squid 177M Oct 11 04:04 access.log.1.gz
-rw-r--r-- 1 squid squid 160M Sep 22 04:04 access.log.20.gz
-rw-r--r-- 1 squid squid 152M Sep 21 04:04 access.log.21.gz
-rw-r--r-- 1 squid squid 163M Sep 20 04:04 access.log.22.gz
-rw-r--r-- 1 squid squid 85M Sep 19 04:09 access.log.23.gz
-rw-r--r-- 1 squid squid 64M Sep 18 04:02 access.log.24.gz
-rw-r--r-- 1 squid squid 127M Sep 17 04:04 access.log.25.gz
-rw-r--r-- 1 squid squid 143M Sep 16 04:04 access.log.26.gz
-rw-r--r-- 1 squid squid 145M Sep 15 04:04 access.log.27.gz
-rw-r--r-- 1 squid squid 143M Sep 14 04:04 access.log.28.gz
-rw-r--r-- 1 squid squid 135M Sep 13 04:03 access.log.29.gz
-rw-r--r-- 1 squid squid 100M Oct 10 04:03 access.log.2.gz
-rw-r--r-- 1 squid squid 63M Sep 12 04:03 access.log.30.gz
-rw-r--r-- 1 squid squid 87M Oct 9 04:03 access.log.3.gz
-rw-r--r-- 1 squid squid 144M Oct 8 04:04 access.log.4.gz
-rw-r--r-- 1 squid squid 157M Oct 7 04:04 access.log.5.gz
-rw-r--r-- 1 squid squid 169M Oct 6 04:05 access.log.6.gz
-rw-r--r-- 1 squid squid 158M Oct 5 04:04 access.log.7.gz
-rw-r--r-- 1 squid squid 176M Oct 4 04:04 access.log.8.gz
-rw-r--r-- 1 squid squid 80M Oct 3 04:03 access.log.9.gz
-rw-r--r-- 1 squid squid 34K Oct 11 14:08 cache.log
-rw-r--r-- 1 squid squid 8.5K Oct 2 04:02 cache.log.10.gz
-rw-r--r-- 1 squid squid 12K Oct 1 04:04 cache.log.11.gz
-rw-r--r-- 1 squid squid 19K Sep 30 04:04 cache.log.12.gz
-rw-r--r-- 1 squid squid 31K Sep 29 04:04 cache.log.13.gz
-rw-r--r-- 1 squid squid 17K Sep 28 04:04 cache.log.14.gz
-rw-r--r-- 1 squid squid 21K Sep 27 04:04 cache.log.15.gz
-rw-r--r-- 1 squid squid 13K Sep 26 04:03 cache.log.16.gz
-rw-r--r-- 1 squid squid 14K Sep 25 04:03 cache.log.17.gz
-rw-r--r-- 1 squid squid 27K Sep 24 04:04 cache.log.18.gz
-rw-r--r-- 1 squid squid 20K Sep 23 04:04 cache.log.19.gz
-rw-r--r-- 1 squid squid 15K Oct 11 04:04 cache.log.1.gz
-rw-r--r-- 1 squid squid 20K Sep 22 04:04 cache.log.20.gz
-rw-r--r-- 1 squid squid 18K Sep 21 04:04 cache.log.21.gz
-rw-r--r-- 1 squid squid 14K Sep 20 04:04 cache.log.22.gz
-rw-r--r-- 1 squid squid 12K Sep 19 04:09 cache.log.23.gz
-rw-r--r-- 1 squid squid 8.6K Sep 18 04:02 cache.log.24.gz
-rw-r--r-- 1 squid squid 17K Sep 17 04:04 cache.log.25.gz
-rw-r--r-- 1 squid squid 19K Sep 16 04:04 cache.log.26.gz
-rw-r--r-- 1 squid squid 22K Sep 15 04:04 cache.log.27.gz
-rw-r--r-- 1 squid squid 17K Sep 14 04:04 cache.log.28.gz
-rw-r--r-- 1 squid squid 17K Sep 13 04:03 cache.log.29.gz
-rw-r--r-- 1 squid squid 7.8K Oct 10 04:03 cache.log.2.gz
-rw-r--r-- 1 squid squid 12K Sep 12 04:03 cache.log.30.gz
-rw-r--r-- 1 squid squid 9.1K Oct 9 04:03 cache.log.3.gz
-rw-r--r-- 1 squid squid 10K Oct 8 04:04 cache.log.4.gz
-rw-r--r-- 1 squid squid 13K Oct 7 04:04 cache.log.5.gz
-rw-r--r-- 1 squid squid 16K Oct 6 04:05 cache.log.6.gz
-rw-r--r-- 1 squid squid 11K Oct 5 04:04 cache.log.7.gz
-rw-r--r-- 1 squid squid 17K Oct 4 04:04 cache.log.8.gz
-rw-r--r-- 1 squid squid 9.3K Oct 3 04:03 cache.log.9.gz
-rw-r--r-- 1 root root 2.2K Oct 11 08:08 squid.out
-rw-r--r-- 1 squid squid 395M Oct 11 14:28 store.log
-rw-r--r-- 1 squid squid 85M Oct 2 04:03 store.log.10.gz
-rw-r--r-- 1 squid squid 235M Oct 1 04:06 store.log.11.gz
-rw-r--r-- 1 squid squid 282M Sep 30 04:07 store.log.12.gz
-rw-r--r-- 1 squid squid 285M Sep 29 04:06 store.log.13.gz
-rw-r--r-- 1 squid squid 255M Sep 28 04:06 store.log.14.gz
-rw-r--r-- 1 squid squid 287M Sep 27 04:06 store.log.15.gz
-rw-r--r-- 1 squid squid 157M Sep 26 04:04 store.log.16.gz
-rw-r--r-- 1 squid squid 117M Sep 25 04:04 store.log.17.gz
-rw-r--r-- 1 squid squid 246M Sep 24 04:06 store.log.18.gz
-rw-r--r-- 1 squid squid 262M Sep 23 04:07 store.log.19.gz
-rw-r--r-- 1 squid squid 296M Oct 11 04:07 store.log.1.gz
-rw-r--r-- 1 squid squid 264M Sep 22 04:06 store.log.20.gz
-rw-r--r-- 1 squid squid 252M Sep 21 04:06 store.log.21.gz
-rw-r--r-- 1 squid squid 273M Sep 20 04:06 store.log.22.gz
-rw-r--r-- 1 squid squid 149M Sep 19 04:10 store.log.23.gz
-rw-r--r-- 1 squid squid 113M Sep 18 04:03 store.log.24.gz
-rw-r--r-- 1 squid squid 213M Sep 17 04:05 store.log.25.gz
-rw-r--r-- 1 squid squid 236M Sep 16 04:06 store.log.26.gz
-rw-r--r-- 1 squid squid 237M Sep 15 04:06 store.log.27.gz
-rw-r--r-- 1 squid squid 233M Sep 14 04:06 store.log.28.gz
-rw-r--r-- 1 squid squid 221M Sep 13 04:05 store.log.29.gz
-rw-r--r-- 1 squid squid 178M Oct 10 04:05 store.log.2.gz
-rw-r--r-- 1 squid squid 108M Sep 12 04:03 store.log.30.gz
-rw-r--r-- 1 squid squid 155M Oct 9 04:04 store.log.3.gz
-rw-r--r-- 1 squid squid 239M Oct 8 04:06 store.log.4.gz
-rw-r--r-- 1 squid squid 263M Oct 7 04:07 store.log.5.gz
-rw-r--r-- 1 squid squid 280M Oct 6 04:07 store.log.6.gz
-rw-r--r-- 1 squid squid 261M Oct 5 04:06 store.log.7.gz
-rw-r--r-- 1 squid squid 293M Oct 4 04:07 store.log.8.gz
-rw-r--r-- 1 squid squid 141M Oct 3 04:04 store.log.9.gz
>From: trainier@xxxxxxxxxx
>To: squid-users@xxxxxxxxxxxxxxx
>Subject: Re: Crashed squid 2.5.STABLE11
>Date: Tue, 11 Oct 2005 11:28:56 -0400
>
>Does this file exist? -> /var/log/squid/store.log
>Does the user running squid have permission to write to it?
>
>Basically, do an ls -lah /var/log/squid
>and paste the output into the reply email.
>
>
>
>Tim Rainier
>Information Services, Kalsec, INC
>trainier@xxxxxxxxxx
>
>"Lucia Di Occhi" <saint_lucy@xxxxxxxxxxx> wrote on 10/11/2005 10:21:52
AM:
>
> > Linux squid 2.6.12-1.1378_FC3smp #1 SMP Wed Sep 14 04:52:36 EDT 2005
>i686
> > i686 i386 GNU/Linux
> >
> > Squid Cache: Version 2.5.STABLE11
> > configure options: --program-prefix= --prefix=/usr --exec-prefix=/usr
> > --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
>--datadir=/usr/share
> > --includedir=/usr/include --libdir=/usr/lib --libexecdir=/usr/libexec
> > --localstatedir=/var --sharedstatedir=/usr/com --mandir=/usr/share/man
> > --infodir=/usr/share/info --exec_prefix=/usr --bindir=/usr/sbin
> > --libexecdir=/usr/lib/squid --localstatedir=/var
--sysconfdir=/etc/squid
>
> > --enable-poll --enable-removal-policies=heap,lru
> > --enable-storeio=aufs,coss,diskd,null,ufs --enable-linux-netfilter
> > --with-pthreads --enable-useragent-log --enable-referer-log
> > --disable-dependency-tracking --enable-cachemgr-hostname=localhost
> > --disable-ident-lookups --enable-truncate --enable-underscores
> > --datadir=/usr/share
> >
> > Reading specs from /usr/lib/gcc/i386-redhat-linux/3.4.4/specs
> > Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
> > --infodir=/usr/share/info --enable-shared --enable-threads=posix
> > --disable-checking --with-system-zlib --enable-__cxa_atexit
> > --disable-libunwind-exceptions --enable-java-awt=gtk
> > --host=i386-redhat-linux
> > Thread model: posix
> > gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)
> >
> > From cache.log
> >
> > 2005/10/11 02:31:33| WARNING: unparseable HTTP header field
> > {403,14,URL,/vanityParser.cfm}
> > 2005/10/11 02:31:33| WARNING: unparseable HTTP header field
> > {404,*,URL,/test.htm}
> > 2005/10/11 02:31:33| WARNING: unparseable HTTP header field
> > {500,100,URL,/iisHelp/common/500-100.asp}
> > 2005/10/11 02:31:33| ctx: exit level 0
> > 2005/10/11 02:31:33| WARNING: unparseable HTTP header field
> > {403,14,URL,/vanityParser.cfm}
> > 2005/10/11 02:31:33| WARNING: unparseable HTTP header field
> > {404,*,URL,/test.htm}
> > 2005/10/11 02:31:33| WARNING: unparseable HTTP header field
> > {500,100,URL,/iisHelp/common/500-100.asp}
> > 2005/10/11 02:49:09| httpReadReply: Excess data from "GET
> > http://hpdjjs.com/cgi-bin/jumpstation.py?query=overlandserver";
> > 2005/10/11 03:04:53| httpReadReply: Excess data from "HEAD
> > http://www.overture.com/";
> > 2005/10/11 03:23:51| ctx: enter level 0:
> > 'http://viewmorepics.myspace.com/index.cfm?
> >
>fuseaction=viewImage&friendID=28636111&imageID=251293587&Mytoken=8B0D8DCC-7FD4-4333-991911DE8DEF62D12839554968'
> > 2005/10/11 03:23:51| WARNING: unparseable HTTP header field
> > {403,14,URL,/vanityParser.cfm}
> > 2005/10/11 03:23:51| WARNING: unparseable HTTP header field
> > {404,*,URL,/test.htm}
> > 2005/10/11 03:23:51| WARNING: unparseable HTTP header field
> > {500,100,URL,/iisHelp/common/500-100.asp}
> > 2005/10/11 03:23:51| ctx: exit level 0
> > 2005/10/11 03:23:51| WARNING: unparseable HTTP header field
> > {403,14,URL,/vanityParser.cfm}
> > 2005/10/11 03:23:51| WARNING: unparseable HTTP header field
> > {404,*,URL,/test.htm}
> > 2005/10/11 03:23:51| WARNING: unparseable HTTP header field
> > {500,100,URL,/iisHelp/common/500-100.asp}
> > 2005/10/11 03:29:59| httpReadReply: Excess data from "HEAD
> > http://www.overture.com/";
> > 2005/10/11 04:02:53| storeDirWriteCleanLogs: Starting...
> > 2005/10/11 04:02:53| WARNING: Closing open FD 12
> > 2005/10/11 04:02:54| 65536 entries written so far.
> > 2005/10/11 04:02:54| 131072 entries written so far.
> > 2005/10/11 04:02:54| 196608 entries written so far.
> > 2005/10/11 04:02:54| 262144 entries written so far.
> > 2005/10/11 04:02:54| 327680 entries written so far.
> > 2005/10/11 04:02:54| 393216 entries written so far.
> > 2005/10/11 04:02:54| 458752 entries written so far.
> > 2005/10/11 04:02:54| 524288 entries written so far.
> > 2005/10/11 04:02:54| 589824 entries written so far.
> > 2005/10/11 04:02:54| 655360 entries written so far.
> > 2005/10/11 04:02:54| 720896 entries written so far.
> > 2005/10/11 04:02:54| 786432 entries written so far.
> > 2005/10/11 04:02:54| 851968 entries written so far.
> > 2005/10/11 04:02:54| 917504 entries written so far.
> > 2005/10/11 04:02:54| 983040 entries written so far.
> > 2005/10/11 04:02:54| 1048576 entries written so far.
> > 2005/10/11 04:02:54| 1114112 entries written so far.
> > 2005/10/11 04:02:54| 1179648 entries written so far.
> > 2005/10/11 04:02:54| 1245184 entries written so far.
> > 2005/10/11 04:02:54| 1310720 entries written so far.
> > 2005/10/11 04:02:55| 1376256 entries written so far.
> > 2005/10/11 04:02:55| 1441792 entries written so far.
> > 2005/10/11 04:02:55| 1507328 entries written so far.
> > 2005/10/11 04:02:55| 1572864 entries written so far.
> > 2005/10/11 04:02:55| 1638400 entries written so far.
> > 2005/10/11 04:02:55| 1703936 entries written so far.
> > 2005/10/11 04:02:55| 1769472 entries written so far.
> > 2005/10/11 04:02:55| 1835008 entries written so far.
> > 2005/10/11 04:02:55| 1900544 entries written so far.
> > 2005/10/11 04:02:55| 1966080 entries written so far.
> > 2005/10/11 04:02:55| 2031616 entries written so far.
> > 2005/10/11 04:02:55| 2097152 entries written so far.
> > 2005/10/11 04:02:55| 2162688 entries written so far.
> > 2005/10/11 04:02:56| 2228224 entries written so far.
> > 2005/10/11 04:02:56| 2293760 entries written so far.
> > 2005/10/11 04:02:56| 2359296 entries written so far.
> > 2005/10/11 04:02:56| 2424832 entries written so far.
> > 2005/10/11 04:02:56| 2490368 entries written so far.
> > 2005/10/11 04:02:56| 2555904 entries written so far.
> > 2005/10/11 04:02:56| 2621440 entries written so far.
> > 2005/10/11 04:02:56| 2686976 entries written so far.
> > 2005/10/11 04:02:56| 2752512 entries written so far.
> > 2005/10/11 04:02:57| 2818048 entries written so far.
> > 2005/10/11 04:02:57| 2883584 entries written so far.
> > 2005/10/11 04:03:00| Finished. Wrote 2885180 entries.
> > 2005/10/11 04:03:00| Took 6.3 seconds (459816.8 entries/sec).
> > FATAL: logfileWrite: /var/log/squid/store.log: (0) Success
> >
> > Squid Cache (Version 2.5.STABLE11): Terminated abnormally.
> > CPU Usage: 52558.798 seconds = 17715.523 user + 34843.275 sys
> > Maximum Resident Size: 0 KB
> > Page faults with physical i/o: 0
> > Memory usage for squid via mallinfo():
> > total space in arena: 1616044 KB
> > Ordinary blocks: 1589468 KB 289245 blks
> > Small blocks: 0 KB 0 blks
> > Holding blocks: 3284 KB 7 blks
> > Free Small blocks: 0 KB
> > Free Ordinary blocks: 26575 KB
> > Total in use: 1592752 KB 99%
> > Total free: 26575 KB 2%
> > 2005/10/11 04:03:07| Starting Squid Cache version 2.5.STABLE11 for
> > i686-pc-linux-gnu...
> > ...
> > ...
> > ...
> >
> > squid[/tmp]tail -10000 access.log.1 | grep overture
> > 1129017007.008 455 127.0.0.1 TCP_MISS/200 13106 GET
> > http://www.thesearchster.com/cgi-bin/overture/allsize.cgi? -
> > DIRECT/209.225.11.243 img/png
> > 1129017563.877 222 127.0.0.1 TCP_MISS/200 3633 GET
> > http://cmhtml.vendare.overture.com/d/search/p/vendare/js/ctxt/? -
> > DIRECT/64.209.232.207 application/x-javascript
> > 1129017763.617 384 127.0.0.1 TCP_MISS/200 5845 GET
> > http://ypn-js.overture.com/partner/js/ypn.js - DIRECT/63.163.102.61
> > application/x-javascript
> > 1129017764.139 441 127.0.0.1 TCP_MISS/200 4021 GET
> > http://ypn-js.overture.com/d/search/p/ypn/jsads/? -
DIRECT/63.163.102.61
>
> > text/html
> > 1129017764.548 595 127.0.0.1 TCP_MISS/200 6987 GET
> > http://ypn-js.overture.com/d/search/p/ypn/jsads/? -
DIRECT/63.163.102.61
>
> > text/html
> >
> >
> > I hope this helps.
> >
> > _________________________________________________________________
> > Express yourself instantly with MSN Messenger! Download today - it's
>FREE!
> > http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
> >
>
_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today - it's FREE!
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
_________________________________________________________________
Is your PC infected? Get a FREE online computer virus scan from McAfee®
Security. http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963
