Is it 32-bit system?
If yes, i guess with PAE process cannot take more than 2GB of memory. But i
think i am wrong (it is 64-bit?).
try to do sometimes "ps axv", and see how much ram eated by processes
On Tue, 6 Mar 2007 14:47:02 -0500, Dave Rhodes wrote
> Hello All:
>
> I am running Squid 2.6 Stable5 on an HP DL390 w/6GB RAM, 60GB cache,
> 2GB swap w/SuSE 10.1. As a rule, thanks to some help from Henrik,
> everything runs well. Twice now though, I've had oom-killer jump in
> and kill Squid and only Squid. I am running a very small test group
> of about 30 users so it takes awhile (about 3 weeks) to run out of memory.
>
> After the last crash, I put in memory monitors to see what the status
> was just before and just after the crash. I also added Webmin monitors
> to let me know when the process stopped. Just before this particular
> crash, the free memory was only about 15MB but that is not unusual with
> Linux and I had watched it on several occasions jump from that
> number to almost 100MB. Apparently, it ran out of buffers and oom-
> killer shut the Squid process down. After restarting the Squid
> process a check of the free memory showed 5.4GB.
>
> If anyone can tell me why this happens, I would certainly appreciate
> it.
>
> Below is the system log output detailing the oom-killer sequence.
> The access.log and cache.log show no problems at all. Thanks, Dave
>
> System Log:
>
> Mar 6 11:01:10 Squid1 kernel: oom-killer: gfp_mask=0x201d2, order=0
> Mar 6 11:01:10 Squid1 kernel:
> Mar 6 11:01:10 Squid1 kernel: Call Trace:
> <ffffffff8015af7b>{out_of_memory+93}
> <ffffffff8015cde9>{__alloc_pages+552}
> Mar 6 11:01:10 Squid1 kernel:
> <ffffffff8015e617>{__do_page_cache_readahe
> ad+149} <ffffffff802cfa4a>{__wait_on_bit_lock+91}
> Mar 6 11:01:10 Squid1 kernel:
> <ffffffff80157f94>{__lock_page+114} <fffff
> fff8015a59b>{filemap_nopage+323}
> Mar 6 11:01:10 Squid1 kernel:
> <ffffffff80164cc1>{__handle_mm_fault+911}
> <ffffffff802d235e>{do_page_fault+965}
> Mar 6 11:01:10 Squid1 kernel:
> <ffffffff801291aa>{default_wake_function+0
> } <ffffffff8010de94>{do_gettimeofday+80}
>
> Mar 6 11:01:10 Squid1 kernel: <ffffffff8010b451>{error_exit+0}
> Mar 6 11:01:10 Squid1 kernel: Mem-info:
> Mar 6 11:01:10 Squid1 kernel: Node 0 DMA per-cpu:
> Mar 6 11:01:10 Squid1 kernel: cpu 0 hot: high 0, batch 1 used:0 Mar
> 6
> 11:01:10 Squid1 kernel: cpu 0 cold: high 0, batch 1 used:0 Mar 6
> 11:01:10 Squid1 kernel: cpu 1 hot: high 0, batch 1 used:0
> Mar 6 11:01:11 Squid1 squid[5079]: Squid Parent: child process 5081
> exited due to signal 9
> Mar 6 11:01:12 Squid1 kernel: cpu 1 cold: high 0, batch 1 used:0
> Mar 6 11:01:15 Squid1 kernel: Node 0 DMA32 per-cpu:
> Mar 6 11:01:16 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:161
> Mar 6 11:01:17 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:14
> Mar 6 11:01:17 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:158
> Mar 6 11:01:17 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:61
> Mar 6 11:01:17 Squid1 kernel: Node 0 Normal per-cpu:
> Mar 6 11:01:17 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:177
> Mar 6 11:01:17 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:15
> Mar 6 11:01:17 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:156
> Mar 6 11:01:17 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:11
> Mar 6 11:01:17 Squid1 kernel: Node 0 HighMem per-cpu: empty
> Mar 6 11:01:17 Squid1 kernel: Free pages: 32040kB (0kB HighMem)
> Mar 6 11:01:17 Squid1 kernel: Active:1046725 inactive:437196
> dirty:0 writeback: 0 unstable:0
> free:8010 slab:5467 mapped:1480421 pagetables:5102
>
> Mar 6 11:01:17 Squid1 kernel: Node 0 DMA free:12388kB min:16kB low:20kB
> high:24 kB active:0kB
> inactive:0kB present:12032kB pages_scanned:2868 all_unreclaimable?
> yes
> Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 3512 5974 5974
> Mar 6 11:01:17 Squid1 kernel: Node 0 DMA32 free:15616kB min:5808kB
> low:7260kB h igh:8712kB
> active:2228388kB inactive:1308212kB present:3596460kB pages_scanned:5
> 390781 all_unreclaimable? yes
> Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 0 2461 2461
> Mar 6 11:01:17 Squid1 kernel: Node 0 Normal free:4036kB min:4068kB
> low:5084kB h igh:6100kB
> active:1958512kB inactive:440572kB present:2520960kB pages_scanned:98
> 21648 all_unreclaimable? yes
> Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 0 0 0
> Mar 6 11:01:17 Squid1 kernel: Node 0 HighMem free:0kB min:128kB
> low:128kB high: 128kB
> active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 0 0 0
> Mar 6 11:01:17 Squid1 kernel: Node 0 DMA: 7*4kB 5*8kB 4*16kB 5*32kB
> 3*64kB 3*12 8kB 1*256kB
> 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12388kB
> Mar 6 11:01:17 Squid1 kernel: Node 0 DMA32: 2*4kB 1*8kB 1*16kB
> 3*32kB 0*64kB 1* 128kB 0*256kB
> 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15616kB
>
> Mar 6 11:01:17 Squid1 kernel: Node 0 Normal: 1*4kB 0*8kB 0*16kB 2*32kB
> 0*64kB 1 *128kB 1*256kB
> 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4036kB
> Mar 6 11:01:17 Squid1 kernel: Node 0 HighMem: empty
>
> Mar 6 11:01:17 Squid1 kernel: Swap cache: add 1031992, delete
> 1031737, find 761
> 567/818419, race 0+14
> Mar 6 11:01:17 Squid1 kernel: Free swap = 0kB
> Mar 6 11:01:17 Squid1 kernel: Total swap = 2104476kB
> Mar 6 11:01:17 Squid1 kernel: Free swap: 0kB
> Mar 6 11:01:17 Squid1 kernel: 1687551 pages of RAM
> Mar 6 11:01:17 Squid1 kernel: 174888 reserved pages
> Mar 6 11:01:17 Squid1 kernel: 10669 pages shared
> Mar 6 11:01:17 Squid1 kernel: 255 pages swap cached
> Mar 6 11:01:17 Squid1 kernel: oom-killer: gfp_mask=0x201d2, order=0
> Mar 6 11:01:17 Squid1 kernel:
> Mar 6 11:01:17 Squid1 kernel: Call Trace:
<ffffffff8015af7b>{out_of_memory+93}
> <ffffffff8015cde9>{__alloc_pages+552}
>
> Mar 6 11:01:17 Squid1 kernel:
> <ffffffff8012a186>{__wake_up+56} <ffffffff
8015e617>{__do_page_cache_readahead+149}
> Mar 6 11:01:17 Squid1 kernel:
> <ffffffff801e97bf>{find_next_bit+85} <ffff
> ffff80128243>{find_busiest_group+356}
> Mar 6 11:01:17 Squid1 kernel:
> <ffffffff8015a59b>{filemap_nopage+323} <ff
> ffffff80164cc1>{__handle_mm_fault+911}
>
> Mar 6 11:01:17 Squid1 kernel: <ffffffff80143d9d>{finish_wait+50}
> <ffffff
> ff802d235e>{do_page_fault+965}
> Mar 6 11:01:17 Squid1 kernel:
> <ffffffff80143d1a>{autoremove_wake_functio
> n+0} <ffffffff8010b451>{error_exit+0}
> Mar 6 11:01:17 Squid1 kernel: Mem-info:
> Mar 6 11:01:17 Squid1 kernel: Node 0 DMA per-cpu:
> Mar 6 11:01:17 Squid1 kernel: cpu 0 hot: high 0, batch 1 used:0
> Mar 6 11:01:17 Squid1 kernel: cpu 0 cold: high 0, batch 1 used:0
> Mar 6 11:01:17 Squid1 kernel: cpu 1 hot: high 0, batch 1 used:0
> Mar 6 11:01:17 Squid1 kernel: cpu 1 cold: high 0, batch 1 used:0
> Mar 6 11:01:18 Squid1 kernel: Node 0 DMA32 per-cpu:
>
> Mar 6 11:01:18 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:161
> Mar 6 11:01:18 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:14
> Mar 6 11:01:18 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:158
> Mar 6 11:01:18 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:61
> Mar 6 11:01:18 Squid1 kernel: Node 0 Normal per-cpu:
> Mar 6 11:01:18 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:177
> Mar 6 11:01:18 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:15
> Mar 6 11:01:18 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:156
> Mar 6 11:01:18 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:11
> Mar 6 11:01:18 Squid1 kernel: Node 0 HighMem per-cpu: empty
> Mar 6 11:01:18 Squid1 kernel: Free pages: 32040kB (0kB HighMem)
> Mar 6 11:01:18 Squid1 kernel: Active:1046467 inactive:437482
> dirty:0 writeback: 0 unstable:0
> free:8010 slab:5467 mapped:1480421 pagetables:5102
>
> Mar 6 11:01:18 Squid1 kernel: Node 0 DMA free:12388kB min:16kB low:20kB
> high:24 kB active:0kB
> inactive:0kB present:12032kB pages_scanned:2868 all_unreclaimable?
> yes
> Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 3512 5974 5974
> Mar 6 11:01:18 Squid1 kernel: Node 0 DMA32 free:15616kB min:5808kB
> low:7260kB h igh:8712kB
> active:2228060kB inactive:1308540kB present:3596460kB pages_scanned:5
> 440499 all_unreclaimable? yes
> Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 0 2461 2461
> Mar 6 11:01:18 Squid1 kernel: Node 0 Normal free:4036kB min:4068kB
> low:5084kB h igh:6100kB
> active:1957808kB inactive:441276kB present:2520960kB pages_scanned:98
> 21962 all_unreclaimable? yes
> Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 0 0 0
> Mar 6 11:01:18 Squid1 kernel: Node 0 HighMem free:0kB min:128kB
> low:128kB high: 128kB
> active:0kB inactive:0kB present:0kB pages_scanned:0
> all_unreclaimable? no
> Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 0 0 0
> Mar 6 11:01:18 Squid1 kernel: Node 0 DMA: 7*4kB 5*8kB 4*16kB 5*32kB
> 3*64kB 3*12 8kB 1*256kB
> 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12388kB
> Mar 6 11:01:18 Squid1 kernel: Node 0 DMA32: 2*4kB 1*8kB 1*16kB
> 3*32kB 0*64kB 1* 128kB 0*256kB
> 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15616kB
>
> Mar 6 11:01:18 Squid1 kernel: Node 0 Normal: 1*4kB 0*8kB 0*16kB 2*32kB
> 0*64kB 1 *128kB 1*256kB
> 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4036kB
> Mar 6 11:01:18 Squid1 kernel: Node 0 HighMem: empty
>
> Mar 6 11:01:18 Squid1 kernel: Swap cache: add 1031992, delete
> 1031737, find 761
> 567/818419, race 0+14
> Mar 6 11:01:18 Squid1 kernel: Free swap = 0kB
> Mar 6 11:01:18 Squid1 kernel: Total swap = 2104476kB
> Mar 6 11:01:18 Squid1 kernel: Free swap: 0kB
> Mar 6 11:01:18 Squid1 kernel: oom-killer: gfp_mask=0x201d2, order=0
> Mar 6 11:01:18 Squid1 kernel:
> Mar 6 11:01:18 Squid1 kernel: Call Trace:
<ffffffff8015af7b>{out_of_memory+93}
> <ffffffff8015cde9>{__alloc_pages+552}
> Mar 6 11:01:18 Squid1 kernel:
> <ffffffff8015e617>{__do_page_cache_readahe
> ad+149} <ffffffff802ceec0>{thread_return+0}
> Mar 6 11:01:18 Squid1 kernel:
> <ffffffff8015a59b>{filemap_nopage+323} <ff
> ffffff80164cc1>{__handle_mm_fault+911}
> Mar 6 11:01:18 Squid1 kernel:
> <ffffffff802d235e>{do_page_fault+965} <fff
> fffff8018cb56>{__pollwait+0}
>
> Mar 6 11:01:18 Squid1 kernel: <ffffffff8018cb23>{sys_select+795}
> <ffffff
> ff8010b451>{error_exit+0}
> Mar 6 11:01:18 Squid1 kernel: Free swap = 0kB
> Mar 6 11:01:18 Squid1 kernel: Total swap = 2104476kB
> Mar 6 11:01:18 Squid1 kernel: Free swap: 0kB
> Mar 6 11:01:18 Squid1 kernel: 1687551 pages of RAM
> Mar 6 11:01:18 Squid1 kernel: 174888 reserved pages
> Mar 6 11:01:18 Squid1 kernel: 10675 pages shared
> Mar 6 11:01:18 Squid1 kernel: 255 pages swap cached
>
> Mar 6 11:01:18 Squid1 kernel: Out of Memory: Kill process 5081
> (squid) score 29 271 and
> children.
> Mar 6 11:01:18 Squid1 kernel: Out of memory: Killed process 5082
> (unlinkd).
--
Virtual ISP S.A.L.
