why the kmalloc return fail when there is free physical address but return success after dropping page caches

baotiao <baotiao@xxxxxxxxx> · Wed, 18 May 2016 10:38:52 +0800

Hello every, I meet an interesting kernel memory problem. Can anyone help me explain what happen under the kernel
The machine's status is describe as blow:
the machine has 96 physical memory. And the real use memory is about 
64G, and the page cache use about 32G. we also use the swap area, at 
that time we have about 10G(we set the swap max size to 32G). At that 
moment, we find xfs report 

Apr 29 21:54:31 w-openstack86 kernel: XFS: possible memory allocation deadlock in kmem_alloc (mode:0x250)
after reading the source code. This message is display from this line 

    ptr = kmalloc(size, lflags);
    if (ptr || (flags & (KM_MAYFAIL|KM_NOSLEEP)))
      return ptr;
    if (!(++retries % 100))
      xfs_err(NULL,
    "possible memory allocation deadlock in %s (mode:0x%x)",
          __func__, lflags);
    congestion_wait(BLK_RW_ASYNC, HZ/50);
The error is cause by the kmalloc() function, there is not enough memory in the system. But there is still 32G page cache.
So I run 

echo 3 > /proc/sys/vm/drop_caches
to drop the page cache.
Then the system is fine. But I really don't know the reason.
Why after I run drop_caches operation the kmalloc() function will 
success? I think even we use whole physical memory, but we only use 64 
real momory, the 32G memory are page cache, further we have enough swap 
space. So why the kernel don't flush the page cache or the swap to 
reserved the kmalloc operation.

----------------------------------------

Github: https://github.com/baotiao
Blog: http://baotiao.github.io/
Stackoverflow: http://stackoverflow.com/users/634415/baotiao 
Linkedin: http://www.linkedin.com/profile/view?id=145231990