UK SquidUser (AXA-TECH-UK) wrote:
Hi...
we are running squid 2.6 stable 17 that have been built within the last
6 months.
The problem we are experiencing is that one of our servers has hit 100%
on the proxycache partition. It is configured in its own filesystem, it
shares with nothing... it has a partition of 36Gb and squid is
configured in squid.conf to use 28Gb. Once it has got to 28Gb we see the
following messages in cache.log (some entries have been deleted,
hopefully I have the relevant ones below)
Rebuilding storage in /proxycache (DIRTY)
Store rebuilding is 0.1% complete
2009/01/06 12:36:11| diskHandleWrite: FD 536: disk write error: (28) No
space left on device
2009/01/06 12:36:11| storeAufsWriteDone: got failure (-6)
2009/01/06 12:36:11| storeSwapOutFileClosed: dirno 0, swapfile 00001392,
errflag=-6
(28) No space left on device
2009/01/06 12:36:29| WARNING: newer swaplog entry for dirno 0, fileno
000015D8
2009/01/06 12:44:13| Store rebuilding is 92.2% complete
2009/01/06 12:44:15| WARNING: Disk space over limit: 13552 KB > 13512 KB
2009/01/06 12:44:26| WARNING: Disk space over limit: 13552 KB > 13512 KB
Huh. Here Squid is saying that the disk usage is ~13MB.
FATAL: xcalloc: Unable to allocate 1 blocks of 28 bytes!
Squid Cache (Version 2.6.STABLE17): Terminated abnormally.
CPU Usage: 531.367 seconds = 482.302 user + 49.066 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0
Memory usage for squid via mallinfo():
total space in arena: -307440 KB
I don't think that's supposed to be negative...
Ordinary blocks: -308814 KB 87176 blks
Small blocks: 0 KB 0 blks
Holding blocks: 25248 KB 6 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 1373 KB
Total in use: -283566 KB 100%
Total free: 1373 KB 0%
2009/01/06 12:44:27| Not currently OK to rewrite swap log.
2009/01/06 12:44:27| storeDirWriteCleanLogs: Operation aborted.
2009/01/06 12:44:38| Store rebuilding is 0.0% complete
And this process repeats continually. We have taken it out of service,
but previously to that it did look like it was processing requests in
the cache.. ie TCP_HIT's were appearing in the log. But surely what is
appearing in the cache log is incorrect.
Squid is configured with the default cache_swap_low and cache_swap_high
and cache_mem 1024 MB, maximum_object_size 16384 KB
maximum_object_size_in_memory 64 KB
cache_replacement_policy heap GDSF
memory_replacement_policy heap GDSF
cache_dir aufs /proxycache 28000 256 256
Can anyone advise why the proxycache hits 100% and it appears to have
problems when it does?
My first guess would be either swap.state or filesystem corruption.
Find and remove the swap.state files (if you didn't specify a location
in squid.conf, it should reside in /proxycache) and see if that fixes
it. If that doesn't do it, reformat /proxycache, run squid -z to
rebuild the directory structure and start it up.
Thanks, K.
Kev Shurmer
Network Analyst - TS Data Networks
AXA Technology Services
kev.shurmer@xxxxxxxxxxxx
Tel. : +44 1 253 68 4652 - Mob. : +44 7974 83 0090
Chris
