just had a server hang on me...seems pretty clearly that some process
stole all the RAM (clamd?)
Jul 30 16:26:04 srv1 kernel: auditd invoked oom-killer:
gfp_mask=0x201d2, order=0, oomkilladj=-17
Jul 30 16:26:08 srv1 kernel: [<c0457d36>] out_of_memory+0x72/0x1a4
Jul 30 16:26:08 srv1 kernel: [<c0459161>] __alloc_pages+0x201/0x282
Jul 30 16:26:08 srv1 kernel: [<c045a3bf>] __do_page_cache_readahead
+0xc4/0x1c6
Jul 30 16:26:08 srv1 kernel: [<c0438ecf>] wake_futex+0x3a/0x44
Jul 30 16:26:08 srv1 kernel: [<c043a2a9>] do_futex+0x738/0xb15
Jul 30 16:26:08 srv1 kernel: [<f88d0b96>] dm_any_congested+0x2f/0x35
[dm_mod]
Jul 30 16:26:08 srv1 kernel: [<c04572d8>] filemap_nopage+0x151/0x315
Jul 30 16:26:08 srv1 kernel: [<c045fda3>] __handle_mm_fault+0x172/0x87b
Jul 30 16:26:08 srv1 kernel: [<c0606c2b>] do_page_fault+0x20a/0x4b8
Jul 30 16:26:08 srv1 kernel: [<c0606a21>] do_page_fault+0x0/0x4b8
Jul 30 16:26:08 srv1 kernel: [<c0405a71>] error_code+0x39/0x40
Jul 30 16:26:08 srv1 kernel: =======================
snip...
Jul 30 16:26:21 srv1 kernel: Free pages: 12044kB (124kB HighMem)
Jul 30 16:26:21 srv1 kernel: Active:119791 inactive:120544 dirty:0
writeback:12 unstable:0 free:3011 slab:7550 mapped-file:1746
mapped-anon:215165 pagetabl
es:3364
Jul 30 16:26:32 srv1 kernel: DMA free:4096kB min:68kB low:84kB
high:100kB active:3948kB inactive:3608kB present:16384kB
pages_scanned:17135 all_unreclaimab
le? yes
Jul 30 16:26:32 srv1 kernel: lowmem_reserve[]: 0 0 880 1007
Jul 30 16:26:33 srv1 kernel: DMA32 free:0kB min:0kB low:0kB high:0kB
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable?
no
Jul 30 16:26:33 srv1 kernel: lowmem_reserve[]: 0 0 880 1007
Jul 30 16:26:33 srv1 kernel: Normal free:7824kB min:3756kB low:4692kB
high:5632kB active:413832kB inactive:416972kB present:901120kB
pages_scanned:2000672
all_unreclaimable? yes
Jul 30 16:26:33 srv1 kernel: lowmem_reserve[]: 0 0 0 1023
Jul 30 16:26:33 srv1 kernel: HighMem free:124kB min:128kB low:264kB
high:400kB active:61512kB inactive:61412kB present:130944kB
pages_scanned:271904 all_un
reclaimable? yes
Jul 30 16:26:33 srv1 kernel: lowmem_reserve[]: 0 0 0 0
Jul 30 16:26:33 srv1 kernel: DMA: 0*4kB 0*8kB 24*16kB 4*32kB 0*64kB
0*128kB 2*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 4096kB
Jul 30 16:26:34 srv1 kernel: DMA32: empty
Jul 30 16:26:34 srv1 kernel: Normal: 44*4kB 4*8kB 234*16kB 7*32kB 1*64kB
0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 7824kB
Jul 30 16:26:34 srv1 kernel: HighMem: 3*4kB 0*8kB 3*16kB 0*32kB 1*64kB
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 124kB
Jul 30 16:26:34 srv1 kernel: Swap cache: add 61987202, delete 61986895,
find 48568667/56549915, race 0+19406
Jul 30 16:26:34 srv1 kernel: Free swap = 0kB
Jul 30 16:26:34 srv1 kernel: Total swap = 2031608kB
Jul 30 16:26:34 srv1 kernel: Free swap: 0kB
Jul 30 16:26:34 srv1 kernel: 262112 pages of RAM
Jul 30 16:26:34 srv1 kernel: 32736 pages of HIGHMEM
Jul 30 16:26:34 srv1 kernel: 3327 reserved pages
Jul 30 16:26:34 srv1 kernel: 8186 pages shared
Jul 30 16:26:34 srv1 kernel: 313 pages swap cached
Jul 30 16:26:34 srv1 kernel: 0 pages dirty
Jul 30 16:26:34 srv1 kernel: 12 pages writeback
Jul 30 16:26:34 srv1 kernel: 1746 pages mapped
Jul 30 16:26:34 srv1 kernel: 7550 pages slab
Jul 30 16:26:34 srv1 kernel: 3364 pages pagetables
how does one determine who the culprit was?
Craig
_______________________________________________
CentOS mailing list
CentOS@xxxxxxxxxx
http://lists.centos.org/mailman/listinfo/centos
