On 4/23/19 12:19 AM, Michal Hocko wrote: > On Mon 22-04-19 21:07:28, Mike Kravetz wrote: >> In our distro kernel, I am thinking about making allocations try "less hard" >> on nodes where we start to see failures. less hard == NORETRY/NORECLAIM. >> I was going to try something like this on an upstream kernel when I noticed >> that it seems like direct reclaim may never end/exit. It 'may' exit, but I >> instrumented __alloc_pages_slowpath() and saw it take well over an hour >> before I 'tricked' it into exiting. >> >> [ 5916.248341] hpage_slow_alloc: jiffies 5295742 tries 2 node 0 success >> [ 5916.249271] reclaim 5295741 compact 1 > > This is unexpected though. What does tries mean? Number of reclaim > attempts? If yes could you enable tracing to see what takes so long in > the reclaim path? tries is the number of times we pass the 'retry:' label in __alloc_pages_slowpath. In this specific case, I am pretty sure all that time is in one call to __alloc_pages_direct_reclaim. My 'trick' to make this succeed was to "echo 0 > nr_hugepages" in another shell. >> This is where it stalled after "echo 4096 > nr_hugepages" on a little VM >> with 8GB total memory. >> >> I have not started looking at the direct reclaim code to see exactly where >> we may be stuck, or trying really hard. My question is, "Is this expected >> or should direct reclaim be somewhat bounded?" With __alloc_pages_slowpath >> getting 'stuck' in direct reclaim, the documented behavior for huge page >> allocation is not going to happen. > > Well, our "how hard to try for hugetlb pages" is quite arbitrary. We > used to rety as long as at least order worth of pages have been > reclaimed but that didn't make any sense since the lumpy reclaim was > gone. Yes, that is what I am seeing in our older distro kernel and I can at least deal with that. > So the semantic has change to reclaim&compact as long as there is > some progress. From what I understad above it seems that you are not > thrashing and calling reclaim again and again but rather one reclaim > round takes ages. Correct > That being said, I do not think __GFP_RETRY_MAYFAIL is wrong here. It > looks like there is something wrong in the reclaim going on. Ok, I will start digging into that. Just wanted to make sure before I got into it too deep. BTW - This is very easy to reproduce. Just try to allocate more huge pages than will fit into memory. I see this 'reclaim taking forever' behavior on v5.1-rc5-mmotm-2019-04-19-14-53. Looks like it was there in v5.0 as well. -- Mike Kravetz
