Re: Repeated fork() causes SLAB to grow without bound

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 08/15/2012 10:46 PM, Daniel Forrest wrote:
I'm hoping someone has seen this before...

I've been trying to track down a performance problem with Linux 3.0.4.
The symptom is system-mode load increasing over time while user-mode
load remains constant while running a data ingest/processing program.

Looking at /proc/meminfo I noticed SUnreclaim increasing steadily.

Looking at /proc/slabinfo I noticed anon_vma and anon_vma_chain also
increasing steadily.

Oh dear.

Basically, what happens is that at fork time, a new
"level" is created for the anon_vma hierarchy. This
works great for normal forking daemons, since the
parent process just keeps running, and forking off
children.

Look at anon_vma_fork() in mm/rmap.c for the details.

Having each child become the new parent, and the
previous parent exit, can result in an "infinite"
stack of anon_vmas.

Now, the parent anon_vma we cannot get rid of,
because that is where the anon_vma lock lives.

However, in your case you have many more anon_vma
levels than you have processes!

I wonder if it may be possible to fix your bug
by adding a refcount to the struct anon_vma,
one count for each VMA that is directly attached
to the anon_vma (ie. vma->anon_vma == anon_vma),
and one for each page that points to the anon_vma.

If the reference count on an anon_vma reaches 0,
we can skip that anon_vma in anon_vma_clone, and
the child process should not get that anon_vma.

A scheme like that may be enough to avoid the trouble
you are running into.

Does this sound realistic?

I was able to generate a simple test program that will cause this:

---

#include <unistd.h>

int main(int argc, char *argv[])
{
    pid_t pid;

    while (1) {
       pid = fork();
       if (pid == -1) {
	 /* error */
	 return 1;
       }
       if (pid) {
	 /* parent */
	 sleep(2);
	 break;
       }
       else {
	 /* child */
	 sleep(1);
       }
    }
    return 0;
}

---

In the actual program (running as a daemon), a child is reading data
while its parent is processing the previously read data.  At any time
there are only a few processes in existence, with older processes
exiting and new processes being fork()ed.  Killing the program frees
the slab usage.

I patched the kernel to 3.0.40, but the problem remains.  I also
compiled with slab debugging and can see that the growth of anon_vma
and anon_vma_chain is due to anon_vma_clone/anon_vma_fork.

Is this a known issue?  Is it fixed in a later release?

Thanks,



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>


[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]