On Wed, 13 Feb 2013, Jonathan Woithe wrote:
We have a number of Linux systems in the field controlling some
instrumentation. They are backed by UPS and local generators so uptimes are
often several hundred days. Recently we have noticed that after a few
hundred days of uptime one of these systems started to trigger the OOM
killer repeatedly even though "free" reports plenty of memory free. After
speaking with a few people at linux.conf.au it was thought that an
exhaustion of lowmem may be implicated and the suggestion was to ask here to
see if anyone had ideas as to what the underlying cause might be.
The allocation triggering the oom killer is a standard GFP_KERNEL
allocation, your lowmem is not depleted.
The output of "ps auxw" is practically unchanged from the time the machine
boots.
Some system specifications:
- CPU: i7 860 at 2.8 GHz
- Mainboard: Advantech AIMB-780
- RAM: 4 GB
- Kernel: 2.6.35.11 SMP, 32 bit (kernel.org kernel, no patches applied)
I'm afraid you're not going to get much help with a 2 1/2 year old kernel.
The machine is set up to boot during which time the data acquisition program
is started. The system then runs indefinitely. Beyond the data acquisition
program the machine does very little. The memory usage of all processes on
the system (including the data acquisition program) as reported by "ps"
remains unchanged for the lifetime of the processes.
This isn't surprising, the vast majority of RAM is consumed by slab.
I have however been able to obtain the slabinfo from another machine
operating in a similar way to the one which faulted. Its uptime is just
over 100 days and has not yet started misbehaving. I have noted that the
active_objs and num_objs fields for the kmalloc-32, kmalloc-64 and
kmalloc-128 lines in particular are very large and seem to be increasing
without limit over both short and long timescales (eg: over an hour, the
kmalloc-128 active_objs increased from 2093227 to 2094021). If the
acquisition is stopped (ie: software idling, hardware not sending data to
ethernet port) these statistics do not appear to increase over a 30 minute
period. I do not know whether this is significant.
You're exactly right, and this is what is causing your oom condition.
The first OOM report (about 3 days before we were made aware of the problem):
kernel: ftp invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0
kernel: Pid: 22217, comm: ftp Not tainted 2.6.35.11-smp #2
...
kernel: DMA free:3480kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:12kB inactive_file:52kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15800kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:148kB slab_unreclaimable:12148kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:151 all_unreclaimable? yes
kernel: lowmem_reserve[]: 0 865 2991 2991
kernel: Normal free:3856kB min:3728kB low:4660kB high:5592kB active_anon:0kB inactive_anon:0kB active_file:916kB inactive_file:1068kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:885944kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:7572kB slab_unreclaimable:797376kB kernel_stack:3136kB pagetables:36kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:3616 all_unreclaimable? yes
kernel: lowmem_reserve[]: 0 0 17014 17014
This allocation cannot allocate in highmem, it needs to be allocated from
ZONE_NORMAL above. Notice how your free watermark, 3856KB, is below the
min watermark, 3728KB. This indicates you've simply exhausted the amount
of memory on the system.
Notice also that the amount of RAM this zone has is 865MB and over 90% of
it is slab.
slabinfo after 106 days uptime and continuous operation:
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
...
kmalloc-128 1234556 1235168 128 32 1 : tunables 0 0 0 : slabdata 38599 38599 0
kmalloc-64 1238117 1238144 64 64 1 : tunables 0 0 0 : slabdata 19346 19346 0
kmalloc-32 1236600 1236608 32 128 1 : tunables 0 0 0 : slabdata 9661 9661 0
You've most certainly got a memory leak here and it's surprising to see it
over three separate slab caches.
Any investigation that we could do into that leak would at best result in
a kernel patch to your 2.6.35 kernel; I'm not sure if there is a fix for a
memory leak that matches your symptoms between 2.6.35.11 and 2.6.35.14.
Better yet would be to try to upgrade these machines to a more recent
kernel to see if it is already fixed. Are we allowed to upgrade or at
least enable kmemleak?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>