On Thu, Feb 20, 2020 at 11:09 AM Chris Murphy <lists@xxxxxxxxxxxxxxxxx> wrote: > > On Thu, Feb 20, 2020 at 10:16 AM Luigi Semenzato <semenzato@xxxxxxxxxx> wrote: > > > > I think this is the right group for the memory issues. > > > > I suspect that the problem with failed allocations (ENOMEM) boils down > > to the unreliability of the page allocator. In my experience, under > > pressure (i.e. pages must be swapped out to be reclaimed) allocations > > can fail even when in theory they should succeed. (I wish I were > > wrong and that someone would convincingly correct me.) > > What is vm.swappiness set to on your system? A fellow Fedora > contributor who has consistently reproduced what you describe, has > discovered he has vm.swappiness=0, and even if it's set to 1, the > problem no longer happens. And this is not a documented consequence of > using a value of 0. I am using the default value of 60. A zero value should cause all file pages to be discarded before any anonymous pages are swapped. I wonder if the fellow Fedora contributor's workload has a lot of file pages, so that discarding them is enough for the image allocator to succeed. In that case "sync; echo 1 > /proc/sys/vm/drop_caches" would be a better way of achieving the same result. (By the way, in my experiments I do that just before hibernating.) > > I have a workaround in which I use memcgroups to free pages before > > starting hibernation. The cgroup request "echo $limit > > > .../memory.limit_in_bytes" blocks until memory usage in the chosen > > cgroup is below $limit. However, I have seen this request fail even > > when there is extra available swap space. > > > > The callback for the operation is mem_cgroup_resize_limit() (BTW I am > > looking at kernel version 4.3.5) and that code has a loop where > > try_to_free_pages() is called up to retry_count, which is at least 5. > > Why 5? One suspects that the writer of that code must have also > > realized that the page freeing request is unreliable and it's worth > > trying multiple times. > > > > So you could try something similar. I don't know if there are > > interfaces to try_to_free_pages() other than those in cgroups. If > > not, and you aren't using cgroups, one way might be to start several > > memory-eating processes (such as "dd if=/dev/zero bs=1G count=1 | > > sleep infinity") and monitor allocation, then when they use more than > > 50% of RAM kill them and immediately hibernate before the freed pages > > are reused. If you can build your custom kernel, maybe it's worth > > adding a sysfs entry to invoke try_to_free_pages(). You could also > > change the hibernation code to do that, but having the user-level hook > > may be more flexible. > > Fedora 31+ now uses cgroupsv2. In any case, my use case is making sure > this works correctly, sanely, with mainline kernels because Fedora > doesn't do custom things with the kernel. > > > > -- > Chris Murphy
