Denys, Thanks for the reply, the system is 64 bit. Right now, "ps axv" shows squid as the largest memory user at almost 600MB and growing at about 2MB/min (makes sense with the 3 week crash time). Memory leak maybe? Dave -----Original Message----- From: Denys [mailto:nuclearcat@xxxxxxxxxxxxxx] Sent: Tuesday, March 06, 2007 3:32 PM To: squid-users@xxxxxxxxxxxxxxx Subject: Re: Oom-killer and Squid Is it 32-bit system? If yes, i guess with PAE process cannot take more than 2GB of memory. But i think i am wrong (it is 64-bit?). try to do sometimes "ps axv", and see how much ram eated by processes On Tue, 6 Mar 2007 14:47:02 -0500, Dave Rhodes wrote > Hello All: > > I am running Squid 2.6 Stable5 on an HP DL390 w/6GB RAM, 60GB cache, > 2GB swap w/SuSE 10.1. As a rule, thanks to some help from Henrik, > everything runs well. Twice now though, I've had oom-killer jump in > and kill Squid and only Squid. I am running a very small test group > of about 30 users so it takes awhile (about 3 weeks) to run out of memory. > > After the last crash, I put in memory monitors to see what the status > was just before and just after the crash. I also added Webmin > monitors to let me know when the process stopped. Just before this > particular crash, the free memory was only about 15MB but that is not > unusual with Linux and I had watched it on several occasions jump from > that number to almost 100MB. Apparently, it ran out of buffers and > oom- killer shut the Squid process down. After restarting the Squid > process a check of the free memory showed 5.4GB. > > If anyone can tell me why this happens, I would certainly appreciate > it. > > Below is the system log output detailing the oom-killer sequence. > The access.log and cache.log show no problems at all. Thanks, Dave > > System Log: > > Mar 6 11:01:10 Squid1 kernel: oom-killer: gfp_mask=0x201d2, order=0 > Mar 6 11:01:10 Squid1 kernel: > Mar 6 11:01:10 Squid1 kernel: Call Trace: > <ffffffff8015af7b>{out_of_memory+93} > <ffffffff8015cde9>{__alloc_pages+552} > Mar 6 11:01:10 Squid1 kernel: > <ffffffff8015e617>{__do_page_cache_readahe > ad+149} <ffffffff802cfa4a>{__wait_on_bit_lock+91} > Mar 6 11:01:10 Squid1 kernel: <ffffffff80157f94>{__lock_page+114} > <fffff > fff8015a59b>{filemap_nopage+323} > Mar 6 11:01:10 Squid1 kernel: > <ffffffff80164cc1>{__handle_mm_fault+911} > <ffffffff802d235e>{do_page_fault+965} > Mar 6 11:01:10 Squid1 kernel: > <ffffffff801291aa>{default_wake_function+0 > } <ffffffff8010de94>{do_gettimeofday+80} > > Mar 6 11:01:10 Squid1 kernel: <ffffffff8010b451>{error_exit+0} > Mar 6 11:01:10 Squid1 kernel: Mem-info: > Mar 6 11:01:10 Squid1 kernel: Node 0 DMA per-cpu: > Mar 6 11:01:10 Squid1 kernel: cpu 0 hot: high 0, batch 1 used:0 Mar > 6 > 11:01:10 Squid1 kernel: cpu 0 cold: high 0, batch 1 used:0 Mar 6 > 11:01:10 Squid1 kernel: cpu 1 hot: high 0, batch 1 used:0 > Mar 6 11:01:11 Squid1 squid[5079]: Squid Parent: child process 5081 > exited due to signal 9 > Mar 6 11:01:12 Squid1 kernel: cpu 1 cold: high 0, batch 1 used:0 > Mar 6 11:01:15 Squid1 kernel: Node 0 DMA32 per-cpu: > Mar 6 11:01:16 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:161 > Mar 6 11:01:17 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:14 > Mar 6 11:01:17 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:158 > Mar 6 11:01:17 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:61 > Mar 6 11:01:17 Squid1 kernel: Node 0 Normal per-cpu: > Mar 6 11:01:17 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:177 > Mar 6 11:01:17 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:15 > Mar 6 11:01:17 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:156 > Mar 6 11:01:17 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:11 > Mar 6 11:01:17 Squid1 kernel: Node 0 HighMem per-cpu: empty > Mar 6 11:01:17 Squid1 kernel: Free pages: 32040kB (0kB HighMem) > Mar 6 11:01:17 Squid1 kernel: Active:1046725 inactive:437196 > dirty:0 writeback: 0 unstable:0 > free:8010 slab:5467 mapped:1480421 pagetables:5102 > > Mar 6 11:01:17 Squid1 kernel: Node 0 DMA free:12388kB min:16kB low:20kB > high:24 kB active:0kB > inactive:0kB present:12032kB pages_scanned:2868 all_unreclaimable? yes > Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 3512 5974 5974 > Mar 6 11:01:17 Squid1 kernel: Node 0 DMA32 free:15616kB min:5808kB > low:7260kB h igh:8712kB > active:2228388kB inactive:1308212kB present:3596460kB pages_scanned:5 > 390781 all_unreclaimable? yes > Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 0 2461 2461 > Mar 6 11:01:17 Squid1 kernel: Node 0 Normal free:4036kB min:4068kB > low:5084kB h igh:6100kB > active:1958512kB inactive:440572kB present:2520960kB pages_scanned:98 > 21648 all_unreclaimable? yes > Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 0 0 0 > Mar 6 11:01:17 Squid1 kernel: Node 0 HighMem free:0kB min:128kB > low:128kB high: 128kB > active:0kB inactive:0kB present:0kB pages_scanned:0 > all_unreclaimable? no > Mar 6 11:01:17 Squid1 kernel: lowmem_reserve[]: 0 0 0 0 > Mar 6 11:01:17 Squid1 kernel: Node 0 DMA: 7*4kB 5*8kB 4*16kB 5*32kB > 3*64kB 3*12 8kB 1*256kB > 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12388kB > Mar 6 11:01:17 Squid1 kernel: Node 0 DMA32: 2*4kB 1*8kB 1*16kB > 3*32kB 0*64kB 1* 128kB 0*256kB > 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15616kB > > Mar 6 11:01:17 Squid1 kernel: Node 0 Normal: 1*4kB 0*8kB 0*16kB 2*32kB > 0*64kB 1 *128kB 1*256kB > 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4036kB > Mar 6 11:01:17 Squid1 kernel: Node 0 HighMem: empty > > Mar 6 11:01:17 Squid1 kernel: Swap cache: add 1031992, delete > 1031737, find 761 > 567/818419, race 0+14 > Mar 6 11:01:17 Squid1 kernel: Free swap = 0kB > Mar 6 11:01:17 Squid1 kernel: Total swap = 2104476kB > Mar 6 11:01:17 Squid1 kernel: Free swap: 0kB > Mar 6 11:01:17 Squid1 kernel: 1687551 pages of RAM > Mar 6 11:01:17 Squid1 kernel: 174888 reserved pages > Mar 6 11:01:17 Squid1 kernel: 10669 pages shared > Mar 6 11:01:17 Squid1 kernel: 255 pages swap cached > Mar 6 11:01:17 Squid1 kernel: oom-killer: gfp_mask=0x201d2, order=0 > Mar 6 11:01:17 Squid1 kernel: > Mar 6 11:01:17 Squid1 kernel: Call Trace: <ffffffff8015af7b>{out_of_memory+93} > <ffffffff8015cde9>{__alloc_pages+552} > > Mar 6 11:01:17 Squid1 kernel: > <ffffffff8012a186>{__wake_up+56} <ffffffff 8015e617>{__do_page_cache_readahead+149} > Mar 6 11:01:17 Squid1 kernel: <ffffffff801e97bf>{find_next_bit+85} > <ffff > ffff80128243>{find_busiest_group+356} > Mar 6 11:01:17 Squid1 kernel: <ffffffff8015a59b>{filemap_nopage+323} > <ff > ffffff80164cc1>{__handle_mm_fault+911} > > Mar 6 11:01:17 Squid1 kernel: <ffffffff80143d9d>{finish_wait+50} > <ffffff > ff802d235e>{do_page_fault+965} > Mar 6 11:01:17 Squid1 kernel: > <ffffffff80143d1a>{autoremove_wake_functio > n+0} <ffffffff8010b451>{error_exit+0} > Mar 6 11:01:17 Squid1 kernel: Mem-info: > Mar 6 11:01:17 Squid1 kernel: Node 0 DMA per-cpu: > Mar 6 11:01:17 Squid1 kernel: cpu 0 hot: high 0, batch 1 used:0 > Mar 6 11:01:17 Squid1 kernel: cpu 0 cold: high 0, batch 1 used:0 > Mar 6 11:01:17 Squid1 kernel: cpu 1 hot: high 0, batch 1 used:0 > Mar 6 11:01:17 Squid1 kernel: cpu 1 cold: high 0, batch 1 used:0 > Mar 6 11:01:18 Squid1 kernel: Node 0 DMA32 per-cpu: > > Mar 6 11:01:18 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:161 > Mar 6 11:01:18 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:14 > Mar 6 11:01:18 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:158 > Mar 6 11:01:18 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:61 > Mar 6 11:01:18 Squid1 kernel: Node 0 Normal per-cpu: > Mar 6 11:01:18 Squid1 kernel: cpu 0 hot: high 186, batch 31 used:177 > Mar 6 11:01:18 Squid1 kernel: cpu 0 cold: high 62, batch 15 used:15 > Mar 6 11:01:18 Squid1 kernel: cpu 1 hot: high 186, batch 31 used:156 > Mar 6 11:01:18 Squid1 kernel: cpu 1 cold: high 62, batch 15 used:11 > Mar 6 11:01:18 Squid1 kernel: Node 0 HighMem per-cpu: empty > Mar 6 11:01:18 Squid1 kernel: Free pages: 32040kB (0kB HighMem) > Mar 6 11:01:18 Squid1 kernel: Active:1046467 inactive:437482 > dirty:0 writeback: 0 unstable:0 > free:8010 slab:5467 mapped:1480421 pagetables:5102 > > Mar 6 11:01:18 Squid1 kernel: Node 0 DMA free:12388kB min:16kB low:20kB > high:24 kB active:0kB > inactive:0kB present:12032kB pages_scanned:2868 all_unreclaimable? yes > Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 3512 5974 5974 > Mar 6 11:01:18 Squid1 kernel: Node 0 DMA32 free:15616kB min:5808kB > low:7260kB h igh:8712kB > active:2228060kB inactive:1308540kB present:3596460kB pages_scanned:5 > 440499 all_unreclaimable? yes > Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 0 2461 2461 > Mar 6 11:01:18 Squid1 kernel: Node 0 Normal free:4036kB min:4068kB > low:5084kB h igh:6100kB > active:1957808kB inactive:441276kB present:2520960kB pages_scanned:98 > 21962 all_unreclaimable? yes > Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 0 0 0 > Mar 6 11:01:18 Squid1 kernel: Node 0 HighMem free:0kB min:128kB > low:128kB high: 128kB > active:0kB inactive:0kB present:0kB pages_scanned:0 > all_unreclaimable? no > Mar 6 11:01:18 Squid1 kernel: lowmem_reserve[]: 0 0 0 0 > Mar 6 11:01:18 Squid1 kernel: Node 0 DMA: 7*4kB 5*8kB 4*16kB 5*32kB > 3*64kB 3*12 8kB 1*256kB > 0*512kB 1*1024kB 1*2048kB 2*4096kB = 12388kB > Mar 6 11:01:18 Squid1 kernel: Node 0 DMA32: 2*4kB 1*8kB 1*16kB > 3*32kB 0*64kB 1* 128kB 0*256kB > 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15616kB > > Mar 6 11:01:18 Squid1 kernel: Node 0 Normal: 1*4kB 0*8kB 0*16kB 2*32kB > 0*64kB 1 *128kB 1*256kB > 1*512kB 1*1024kB 1*2048kB 0*4096kB = 4036kB > Mar 6 11:01:18 Squid1 kernel: Node 0 HighMem: empty > > Mar 6 11:01:18 Squid1 kernel: Swap cache: add 1031992, delete > 1031737, find 761 > 567/818419, race 0+14 > Mar 6 11:01:18 Squid1 kernel: Free swap = 0kB > Mar 6 11:01:18 Squid1 kernel: Total swap = 2104476kB > Mar 6 11:01:18 Squid1 kernel: Free swap: 0kB > Mar 6 11:01:18 Squid1 kernel: oom-killer: gfp_mask=0x201d2, order=0 > Mar 6 11:01:18 Squid1 kernel: > Mar 6 11:01:18 Squid1 kernel: Call Trace: <ffffffff8015af7b>{out_of_memory+93} > <ffffffff8015cde9>{__alloc_pages+552} > Mar 6 11:01:18 Squid1 kernel: > <ffffffff8015e617>{__do_page_cache_readahe > ad+149} <ffffffff802ceec0>{thread_return+0} > Mar 6 11:01:18 Squid1 kernel: <ffffffff8015a59b>{filemap_nopage+323} > <ff > ffffff80164cc1>{__handle_mm_fault+911} > Mar 6 11:01:18 Squid1 kernel: <ffffffff802d235e>{do_page_fault+965} > <fff > fffff8018cb56>{__pollwait+0} > > Mar 6 11:01:18 Squid1 kernel: <ffffffff8018cb23>{sys_select+795} > <ffffff > ff8010b451>{error_exit+0} > Mar 6 11:01:18 Squid1 kernel: Free swap = 0kB > Mar 6 11:01:18 Squid1 kernel: Total swap = 2104476kB > Mar 6 11:01:18 Squid1 kernel: Free swap: 0kB > Mar 6 11:01:18 Squid1 kernel: 1687551 pages of RAM > Mar 6 11:01:18 Squid1 kernel: 174888 reserved pages > Mar 6 11:01:18 Squid1 kernel: 10675 pages shared > Mar 6 11:01:18 Squid1 kernel: 255 pages swap cached > > Mar 6 11:01:18 Squid1 kernel: Out of Memory: Kill process 5081 > (squid) score 29 271 and > children. > Mar 6 11:01:18 Squid1 kernel: Out of memory: Killed process 5082 > (unlinkd). -- Virtual ISP S.A.L.
