Anyone have any ideas here? On 13.03.2014 [10:01:27 -0700], Nishanth Aravamudan wrote: > There might have been an error in my original mail, so resending... > > On 11.03.2014 [14:06:14 -0700], Nishanth Aravamudan wrote: > > We have seen the following situation on a test system: > > > > 2-node system, each node has 32GB of memory. > > > > 2 gigantic (16GB) pages reserved at boot-time, both of which are > > allocated from node 1. > > > > SLUB notices this: > > > > [ 0.000000] SLUB: Unable to allocate memory from node 1 > > [ 0.000000] SLUB: Allocating a useless per node structure in order to > > be able to continue > > > > After boot, user then did: > > > > echo 24 > /proc/sys/vm/nr_hugepages > > > > And tasks are stuck: > > > > [<c0000000010980b8>] kexec_stack+0xb8/0x8000 > > [<c0000000000144d0>] .__switch_to+0x1c0/0x390 > > [<c0000000001ac708>] .throttle_direct_reclaim.isra.31+0x238/0x2c0 > > [<c0000000001b0b34>] .try_to_free_pages+0xb4/0x210 > > [<c0000000001a2f1c>] .__alloc_pages_nodemask+0x75c/0xb00 > > [<c0000000001eafb0>] .alloc_fresh_huge_page+0x70/0x150 > > [<c0000000001eb2d0>] .set_max_huge_pages.part.37+0x130/0x2f0 > > [<c0000000001eb7c8>] .hugetlb_sysctl_handler_common+0x168/0x180 > > [<c0000000002ae21c>] .proc_sys_call_handler+0xfc/0x120 > > [<c00000000021dcc0>] .vfs_write+0xe0/0x260 > > [<c00000000021e8c8>] .SyS_write+0x58/0xd0 > > [<c000000000009e7c>] syscall_exit+0x0/0x7c > > > > [<c00000004f9334b0>] 0xc00000004f9334b0 > > [<c0000000000144d0>] .__switch_to+0x1c0/0x390 > > [<c0000000001ac708>] .throttle_direct_reclaim.isra.31+0x238/0x2c0 > > [<c0000000001b0b34>] .try_to_free_pages+0xb4/0x210 > > [<c0000000001a2f1c>] .__alloc_pages_nodemask+0x75c/0xb00 > > [<c0000000001eafb0>] .alloc_fresh_huge_page+0x70/0x150 > > [<c0000000001eb2d0>] .set_max_huge_pages.part.37+0x130/0x2f0 > > [<c0000000001eb7c8>] .hugetlb_sysctl_handler_common+0x168/0x180 > > [<c0000000002ae21c>] .proc_sys_call_handler+0xfc/0x120 > > [<c00000000021dcc0>] .vfs_write+0xe0/0x260 > > [<c00000000021e8c8>] .SyS_write+0x58/0xd0 > > [<c000000000009e7c>] syscall_exit+0x0/0x7c > > > > [<c00000004f91f440>] 0xc00000004f91f440 > > [<c0000000000144d0>] .__switch_to+0x1c0/0x390 > > [<c0000000001ac708>] .throttle_direct_reclaim.isra.31+0x238/0x2c0 > > [<c0000000001b0b34>] .try_to_free_pages+0xb4/0x210 > > [<c0000000001a2f1c>] .__alloc_pages_nodemask+0x75c/0xb00 > > [<c0000000001eafb0>] .alloc_fresh_huge_page+0x70/0x150 > > [<c0000000001eb2d0>] .set_max_huge_pages.part.37+0x130/0x2f0 > > [<c0000000001eb54c>] .nr_hugepages_store_common.isra.39+0xbc/0x1b0 > > [<c0000000003662cc>] .kobj_attr_store+0x2c/0x50 > > [<c0000000002b2c2c>] .sysfs_write_file+0xec/0x1c0 > > [<c00000000021dcc0>] .vfs_write+0xe0/0x260 > > [<c00000000021e8c8>] .SyS_write+0x58/0xd0 > > [<c000000000009e7c>] syscall_exit+0x0/0x7c > > > > kswapd1 is also pegged at this point at 100% cpu. > > > > If we go in and manually: > > > > echo 24 > > > /sys/devices/system/node/node0/hugepages/hugepages-16384kB/nr_hugepages > > > > rather than relying on the interleaving allocator from the sysctl, the > > allocation succeeds (and the echo returns immediately). > > > > I think we are hitting the following: > > > > mm/hugetlb.c::alloc_fresh_huge_page_node(): > > > > page = alloc_pages_exact_node(nid, > > htlb_alloc_mask(h)|__GFP_COMP|__GFP_THISNODE| > > __GFP_REPEAT|__GFP_NOWARN, > > huge_page_order(h)); > > > > include/linux/gfp.h: > > > > #define GFP_THISNODE (__GFP_THISNODE | __GFP_NOWARN | __GFP_NORETRY) > > > > and mm/page_alloc.c::__alloc_pages_slowpath(): > > > > /* > > * GFP_THISNODE (meaning __GFP_THISNODE, __GFP_NORETRY and > > * __GFP_NOWARN set) should not cause reclaim since the subsystem > > * (f.e. slab) using GFP_THISNODE may choose to trigger reclaim > > * using a larger set of nodes after it has established that the > > * allowed per node queues are empty and that nodes are > > * over allocated. > > */ > > if (IS_ENABLED(CONFIG_NUMA) && > > (gfp_mask & GFP_THISNODE) == GFP_THISNODE) > > goto nopage; > > > > so we *do* reclaim in this callpath. Under my reading, since node1 is > > exhausted, no matter how much work kswapd1 does, it will never reclaim > > memory from node1 to satisfy a 16M page allocation request (or any > > other, for that matter). > > > > I see the following possible changes/fixes, but am unsure if > > a) my analysis is right > > b) which is best. > > > > 1) Since we did notice early in boot that (in this case) node 1 was > > exhausted, perhaps we should mark it as such there somehow, and if a > > __GFP_THISNODE allocation request comes through on such a node, we > > immediately fallthrough to nopage? > > > > 2) There is the following check > > /* > > * For order > PAGE_ALLOC_COSTLY_ORDER, if __GFP_REPEAT is > > * specified, then we retry until we no longer reclaim any pages > > * (above), or we've reclaimed an order of pages at least as > > * large as the allocation's order. In both cases, if the > > * allocation still fails, we stop retrying. > > */ > > if (gfp_mask & __GFP_REPEAT && pages_reclaimed < (1 << order)) > > return 1; > > > > I wonder if we should add a check to also be sure that the pages we are > > reclaiming, if __GFP_THISNODE is set, are from the right node? > > > > if (gfp_mask & __GFP_THISNODE && the progress we have made is on > > the node requested?) > > > > 3) did_some_progress could be updated to track where the progress is > > occuring, and if we are in __GFP_THISNODE allocation request and we > > didn't make any progress on the correct node, we fail the allocation? > > > > I think this situation could be reproduced (and am working on it) by > > exhausting a NUMA node with 16M hugepages and then using the generic > > RR allocator to ask for more. Other node exhaustion cases probably > > exist, but since we can't swap the hugepages, it seems like the most > > straightforward way to try and reproduce it. > > > > Any thoughts on this? Am I way off base? > > > > Thanks, > > Nish > > > > _______________________________________________ > > Linuxppc-dev mailing list > > Linuxppc-dev@xxxxxxxxxxxxxxxx > > https://lists.ozlabs.org/listinfo/linuxppc-dev -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxx. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>