Hello All:
I am running Squid 2.6 Stable5 on an HP DL390 w/6GB RAM, 60GB cache, 2GB
swap w/SuSE 10.1. As a rule, thanks to some help from Henrik,
everything runs well. Twice now though, I've had oom-killer jump in and
kill Squid and only Squid. I am running a very small test group of
about 30 users so it takes awhile (about 3 weeks) to run out of memory.
After the last crash, I put in memory monitors to see what the status
was just before and just after the crash. I also added Webmin monitors
to let me know when the process stopped. Just before this particular
crash, the free memory was only about 15MB but that is not unusual with
Linux and I had watched it on several occasions jump from that number to
almost 100MB. Apparently, it ran out of buffers and oom-killer shut the
Squid process down. After restarting the Squid process a check of the
free memory showed 5.4GB.
If anyone can tell me why this happens, I would certainly appreciate it.
Below is the system log output detailing the oom-killer sequence. The
access.log and cache.log show no problems at all.
Thanks, Dave
System Log:
Mar 6 11:01:10 Squid1 kernel: oom-killer: gfp_mask=0x201d2, order=0
Mar 6 11:01:10 Squid1 kernel:
Mar 6 11:01:10 Squid1 kernel: Call Trace:
<ffffffff8015af7b>{out_of_memory+93}
<ffffffff8015cde9>{__alloc_pages+552}
Mar 6 11:01:10 Squid1 kernel:
<ffffffff8015e617>{__do_page_cache_readahe
ad+149} <ffffffff802cfa4a>{__wait_on_bit_lock+91}
Mar 6 11:01:10 Squid1 kernel:
<ffffffff80157f94>{__lock_page+114} <fffff
fff8015a59b>{filemap_nopage+323}
Mar 6 11:01:10 Squid1 kernel:
<ffffffff80164cc1>{__handle_mm_fault+911}
<ffffffff802d235e>{do_page_fault+965}
Mar 6 11:01:10 Squid1 kernel:
<ffffffff801291aa>{default_wake_function+0
} <ffffffff8010de94>{do_gettimeofday+80}
Mar 6 11:01:10 Squid1 kernel: <ffffffff8010b451>{error_exit+0}
Mar 6 11:01:10 Squid1 kernel: Mem-info:
Mar 6 11:01:10 Squid1 kernel: Node 0 DMA per-cpu:
Mar 6 11:01:10 Squid1 kernel: cpu 0 hot: high 0, batch 1 used:0 Mar 6
11:01:10 Squid1 kernel: cpu 0 cold: high 0, batch 1 used:0 Mar 6
11:01:10 Squid1 kernel: cpu 1 hot: high 0, batch 1 used:0
Mar 6 11:01:11 Squid1 squid[5079]: Squid Parent: child process 5081
exited due to signal 9
Mar 6 11:01:12 Squid1 kernel: cpu 1 cold: high 0, batch 1 used:0
Mar 6 11:01:15 Squid1 kernel: Node 0 DMA32 per-cpu:
Mar 6 11:01:16 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:161
Mar 6 11:01:17 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:14
Mar 6 11:01:17 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:158
Mar 6 11:01:17 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:61
Mar 6 11:01:17 Squid1 kernel: Node 0 Normal per-cpu:
Mar 6 11:01:17 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:177
Mar 6 11:01:17 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:15
Mar 6 11:01:17 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:156
Mar 6 11:01:17 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:11
Mar 6 11:01:17 Squid1 kernel: Node 0 HighMem per-cpu: empty
Mar 6 11:01:17 Squid1 kernel: Free pages: 32040kB (0kB HighMem)
Mar 6 11:01:17 Squid1 kernel: Active:1046725 inactive:437196 dirty:0
writeback: 0 unstable:0
free:8010 slab:5467 mapped:1480421 pagetables:5102
Mar 6 11:01:17 Squid1 kernel: Node 0 DMA free:12388kB min:16kB low:20kB
high:24 kB active:0kB
inactive:0kB present:12032kB pages_scanned:2868 all_unreclaimable?
yes
Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 3512 5974 5974
Mar 6 11:01:17 Squid1 kernel: Node 0 DMA32 free:15616kB min:5808kB
low:7260kB h igh:8712kB
active:2228388kB inactive:1308212kB present:3596460kB pages_scanned:5
390781 all_unreclaimable? yes
Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 0 2461 2461
Mar 6 11:01:17 Squid1 kernel: Node 0 Normal free:4036kB min:4068kB
low:5084kB h igh:6100kB
active:1958512kB inactive:440572kB present:2520960kB pages_scanned:98
21648 all_unreclaimable? yes
Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 0 0 0
Mar 6 11:01:17 Squid1 kernel: Node 0 HighMem free:0kB min:128kB
low:128kB high: 128kB
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable?
no
Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 0 0 0
Mar 6 11:01:17 Squid1 kernel: Node 0 DMA: 7*4kB 5*8kB 4*16kB 5*32kB
3*64kB 3*12 8kB 1*256kB
0*512kB 1*1024kB 1*2048kB 2*4096kB = 12388kB
Mar 6 11:01:17 Squid1 kernel: Node 0 DMA32: 2*4kB 1*8kB 1*16kB 3*32kB
0*64kB 1* 128kB 0*256kB
0*512kB 1*1024kB 1*2048kB 3*4096kB = 15616kB
Mar 6 11:01:17 Squid1 kernel: Node 0 Normal: 1*4kB 0*8kB 0*16kB 2*32kB
0*64kB 1 *128kB 1*256kB
1*512kB 1*1024kB 1*2048kB 0*4096kB = 4036kB
Mar 6 11:01:17 Squid1 kernel: Node 0 HighMem: empty
Mar 6 11:01:17 Squid1 kernel: Swap cache: add 1031992, delete 1031737,
find 761 567/818419, race
0+14
Mar 6 11:01:17 Squid1 kernel: Free swap = 0kB
Mar 6 11:01:17 Squid1 kernel: Total swap = 2104476kB
Mar 6 11:01:17 Squid1 kernel: Free swap: 0kB
Mar 6 11:01:17 Squid1 kernel: 1687551 pages of RAM
Mar 6 11:01:17 Squid1 kernel: 174888 reserved pages
Mar 6 11:01:17 Squid1 kernel: 10669 pages shared
Mar 6 11:01:17 Squid1 kernel: 255 pages swap cached
Mar 6 11:01:17 Squid1 kernel: oom-killer: gfp_mask=0x201d2, order=0
Mar 6 11:01:17 Squid1 kernel:
Mar 6 11:01:17 Squid1 kernel: Call Trace:
<ffffffff8015af7b>{out_of_memory+93}
<ffffffff8015cde9>{__alloc_pages+552}
Mar 6 11:01:17 Squid1 kernel: <ffffffff8012a186>{__wake_up+56}
<ffffffff
8015e617>{__do_page_cache_readahead+149}
Mar 6 11:01:17 Squid1 kernel:
<ffffffff801e97bf>{find_next_bit+85} <ffff
ffff80128243>{find_busiest_group+356}
Mar 6 11:01:17 Squid1 kernel:
<ffffffff8015a59b>{filemap_nopage+323} <ff
ffffff80164cc1>{__handle_mm_fault+911}
Mar 6 11:01:17 Squid1 kernel: <ffffffff80143d9d>{finish_wait+50}
<ffffff
ff802d235e>{do_page_fault+965}
Mar 6 11:01:17 Squid1 kernel:
<ffffffff80143d1a>{autoremove_wake_functio
n+0} <ffffffff8010b451>{error_exit+0}
Mar 6 11:01:17 Squid1 kernel: Mem-info:
Mar 6 11:01:17 Squid1 kernel: Node 0 DMA per-cpu:
Mar 6 11:01:17 Squid1 kernel: cpu 0 hot: high 0, batch 1 used:0
Mar 6 11:01:17 Squid1 kernel: cpu 0 cold: high 0, batch 1 used:0
Mar 6 11:01:17 Squid1 kernel: cpu 1 hot: high 0, batch 1 used:0
Mar 6 11:01:17 Squid1 kernel: cpu 1 cold: high 0, batch 1 used:0
Mar 6 11:01:18 Squid1 kernel: Node 0 DMA32 per-cpu:
Mar 6 11:01:18 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:161
Mar 6 11:01:18 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:14
Mar 6 11:01:18 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:158
Mar 6 11:01:18 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:61
Mar 6 11:01:18 Squid1 kernel: Node 0 Normal per-cpu:
Mar 6 11:01:18 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:177
Mar 6 11:01:18 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:15
Mar 6 11:01:18 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:156
Mar 6 11:01:18 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:11
Mar 6 11:01:18 Squid1 kernel: Node 0 HighMem per-cpu: empty
Mar 6 11:01:18 Squid1 kernel: Free pages: 32040kB (0kB HighMem)
Mar 6 11:01:18 Squid1 kernel: Active:1046467 inactive:437482 dirty:0
writeback: 0 unstable:0
free:8010 slab:5467 mapped:1480421 pagetables:5102
Mar 6 11:01:18 Squid1 kernel: Node 0 DMA free:12388kB min:16kB low:20kB
high:24 kB active:0kB
inactive:0kB present:12032kB pages_scanned:2868 all_unreclaimable?
yes
Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 3512 5974 5974
Mar 6 11:01:18 Squid1 kernel: Node 0 DMA32 free:15616kB min:5808kB
low:7260kB h igh:8712kB
active:2228060kB inactive:1308540kB present:3596460kB pages_scanned:5
440499 all_unreclaimable? yes
Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 0 2461 2461
Mar 6 11:01:18 Squid1 kernel: Node 0 Normal free:4036kB min:4068kB
low:5084kB h igh:6100kB
active:1957808kB inactive:441276kB present:2520960kB pages_scanned:98
21962 all_unreclaimable? yes
Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 0 0 0
Mar 6 11:01:18 Squid1 kernel: Node 0 HighMem free:0kB min:128kB
low:128kB high: 128kB
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable?
no
Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 0 0 0
Mar 6 11:01:18 Squid1 kernel: Node 0 DMA: 7*4kB 5*8kB 4*16kB 5*32kB
3*64kB 3*12 8kB 1*256kB
0*512kB 1*1024kB 1*2048kB 2*4096kB = 12388kB
Mar 6 11:01:18 Squid1 kernel: Node 0 DMA32: 2*4kB 1*8kB 1*16kB 3*32kB
0*64kB 1* 128kB 0*256kB
0*512kB 1*1024kB 1*2048kB 3*4096kB = 15616kB
Mar 6 11:01:18 Squid1 kernel: Node 0 Normal: 1*4kB 0*8kB 0*16kB 2*32kB
0*64kB 1 *128kB 1*256kB
1*512kB 1*1024kB 1*2048kB 0*4096kB = 4036kB
Mar 6 11:01:18 Squid1 kernel: Node 0 HighMem: empty
Mar 6 11:01:18 Squid1 kernel: Swap cache: add 1031992, delete 1031737,
find 761 567/818419, race
0+14
Mar 6 11:01:18 Squid1 kernel: Free swap = 0kB
Mar 6 11:01:18 Squid1 kernel: Total swap = 2104476kB
Mar 6 11:01:18 Squid1 kernel: Free swap: 0kB
Mar 6 11:01:18 Squid1 kernel: oom-killer: gfp_mask=0x201d2, order=0
Mar 6 11:01:18 Squid1 kernel:
Mar 6 11:01:18 Squid1 kernel: Call Trace:
<ffffffff8015af7b>{out_of_memory+93}
<ffffffff8015cde9>{__alloc_pages+552}
Mar 6 11:01:18 Squid1 kernel:
<ffffffff8015e617>{__do_page_cache_readahe
ad+149} <ffffffff802ceec0>{thread_return+0}
Mar 6 11:01:18 Squid1 kernel:
<ffffffff8015a59b>{filemap_nopage+323} <ff
ffffff80164cc1>{__handle_mm_fault+911}
Mar 6 11:01:18 Squid1 kernel:
<ffffffff802d235e>{do_page_fault+965} <fff
fffff8018cb56>{__pollwait+0}
Mar 6 11:01:18 Squid1 kernel: <ffffffff8018cb23>{sys_select+795}
<ffffff
ff8010b451>{error_exit+0}
Mar 6 11:01:18 Squid1 kernel: Free swap = 0kB
Mar 6 11:01:18 Squid1 kernel: Total swap = 2104476kB
Mar 6 11:01:18 Squid1 kernel: Free swap: 0kB
Mar 6 11:01:18 Squid1 kernel: 1687551 pages of RAM
Mar 6 11:01:18 Squid1 kernel: 174888 reserved pages
Mar 6 11:01:18 Squid1 kernel: 10675 pages shared
Mar 6 11:01:18 Squid1 kernel: 255 pages swap cached
Mar 6 11:01:18 Squid1 kernel: Out of Memory: Kill process 5081 (squid)
score 29 271 and children.
Mar 6 11:01:18 Squid1 kernel: Out of memory: Killed process 5082
(unlinkd).
